Nov 02 14:55:49.396605 ip-10-0-132-27 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 14:55:49.757118 ip-10-0-132-27 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:49.757118 ip-10-0-132-27 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 14:55:49.757118 ip-10-0-132-27 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:49.757118 ip-10-0-132-27 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:49.760889 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760810    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:49.760889 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760889    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760894    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760897    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760903    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760909    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760912    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760916    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760919    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760922    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760924    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760927    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760930    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760932    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760935    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760938    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760941    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760943    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760946    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760948    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:49.760957 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760951    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760954    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760956    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760959    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760962    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760970    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760974    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760976    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760979    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760981    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760983    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760986    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760989    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760992    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760994    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.760997    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761000    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761002    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761005    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:49.761413 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761007    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761009    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761012    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761014    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761018    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761023    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761025    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761028    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761031    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761034    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761036    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761039    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761041    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761044    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761048    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761051    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761053    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761056    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761058    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761061    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:49.761892 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761063    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761066    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761069    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761072    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761075    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761077    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761080    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761082    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761085    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761087    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761090    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761093    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761095    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761098    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761100    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761103    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761106    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761109    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761111    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761115    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:49.762381 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761118    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761120    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761123    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761126    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761128    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761131    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761134    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761137    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761139    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761142    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761145    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761147    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761151    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761153    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761155    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761158    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761161    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761164    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761166    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761169    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:49.762857 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761172    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761175    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761177    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761181    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761184    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761187    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761190    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761970    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761976    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761979    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761983    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761986    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761988    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761992    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761994    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.761997    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762000    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762002    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762005    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:49.763344 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762008    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762011    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762013    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762016    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762019    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762021    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762024    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762026    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762029    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762031    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762034    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762037    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762039    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762042    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762044    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762047    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762050    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762052    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762055    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762057    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:49.763812 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762060    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762062    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762066    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762068    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762071    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762074    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762076    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762079    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762081    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762084    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762087    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762090    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762092    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762096    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762098    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762101    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762104    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762106    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762109    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:49.764283 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762111    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762114    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762117    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762119    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762122    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762125    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762127    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762130    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762132    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762137    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762140    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762143    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762146    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762148    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762151    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762153    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762156    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762158    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762162    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:49.764752 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762164    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762167    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762169    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762172    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762175    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762178    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762181    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762184    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762187    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762189    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762192    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762194    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762197    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762200    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762203    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762205    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762208    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762211    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762213    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762216    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:49.765244 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762218    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762221    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762224    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762226    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762228    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762233    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762237    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762239    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762242    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762245    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762247    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762251    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762254    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762257    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762260    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762262    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762778    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762784    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762789    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762792    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:49.765767 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762794    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762797    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762800    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762803    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762806    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762808    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762812    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762814    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762817    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762820    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762823    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762825    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762828    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762831    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762833    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762836    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762838    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762844    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762846    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762849    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:49.766238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762852    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762855    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762857    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762860    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762863    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762866    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762869    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762871    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762874    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762877    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762880    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762882    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762885    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762887    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762890    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762893    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762895    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762898    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762901    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762903    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:49.766739 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762906    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762909    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762912    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762914    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762917    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762919    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762922    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762925    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762927    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762932    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762935    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762938    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762940    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762943    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762945    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762948    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762951    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762954    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762956    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762959    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:49.767214 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762961    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762964    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762967    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762971    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762975    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762978    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762981    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762984    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762987    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762991    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762994    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.762997    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763000    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763003    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763005    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763008    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763011    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763013    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763015    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:49.767699 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763018    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763020    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763026    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763028    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763031    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763033    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763036    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763039    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763041    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763044    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763046    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763049    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763052    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763054    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763057    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763060    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763063    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763066    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763070    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:49.768145 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763072    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763075    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763077    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.763080    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764465    2577 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764476    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764484    2577 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764489    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764493    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764496    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764501    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764505    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764509    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764512    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764515    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764519    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764522    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764525    2577 flags.go:64] FLAG: --cgroup-root=""
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764528    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764531    2577 flags.go:64] FLAG: --client-ca-file=""
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764534    2577 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764537    2577 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764541    2577 flags.go:64] FLAG: --cluster-domain=""
Nov 02 14:55:49.768607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764544    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764547    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764551    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764554    2577 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764558    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764561    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764564    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764568    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764571    2577 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764574    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764577    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764580    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764583    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764588    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764591    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764594    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764596    2577 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764599    2577 flags.go:64] FLAG: --enable-server="true"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764602    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764608    2577 flags.go:64] FLAG: --event-burst="100"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764611    2577 flags.go:64] FLAG: --event-qps="50"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764614    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764617    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764620    2577 flags.go:64] FLAG: --eviction-hard=""
Nov 02 14:55:49.769142 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764625    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764628    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764631    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764634    2577 flags.go:64] FLAG: --eviction-soft=""
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764637    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764640    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764643    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764646    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764649    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764652    2577 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764655    2577 flags.go:64] FLAG: --feature-gates=""
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764660    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764663    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764667    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764670    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764673    2577 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764676    2577 flags.go:64] FLAG: --help="false"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764679    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764683    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764686    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764689    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764692    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764696    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 14:55:49.769771 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764699    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764701    2577 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764705    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764708    2577 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764711    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764715    2577 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764718    2577 flags.go:64] FLAG: --kube-reserved=""
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764721    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764724    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764727    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764730    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764733    2577 flags.go:64] FLAG: --lock-file=""
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764736    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764739    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764742    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764747    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764750    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764753    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764756    2577 flags.go:64] FLAG: --logging-format="text"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764759    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764763    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764767    2577 flags.go:64] FLAG: --manifest-url=""
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764770    2577 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764775    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764778    2577 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 14:55:49.770330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764782    2577 flags.go:64] FLAG: --max-pods="110"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764785    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764788    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764791    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764794    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764797    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764800    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764803    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764810    2577 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764814    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764816    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764820    2577 flags.go:64] FLAG: --pod-cidr=""
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764823    2577 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764826    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764829    2577 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764832    2577 flags.go:64] FLAG: --port="10250"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764835    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764838    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0f690a810e9b77533"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764842    2577 flags.go:64] FLAG: --qos-reserved=""
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764844    2577 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764848    2577 flags.go:64] FLAG: --register-node="true"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764850    2577 flags.go:64] FLAG: --register-with-taints=""
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764854    2577 flags.go:64] FLAG: --registry-burst="10"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764857    2577 flags.go:64] FLAG: --registry-qps="5"
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764860    2577 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764863    2577 flags.go:64] FLAG: --reserved-memory=""
Nov 02 14:55:49.770923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764867    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764870    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764873    2577 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764876    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764879    2577 flags.go:64] FLAG: --runonce="false"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764882    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764885    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764888    2577 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764891    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764894    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764898    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764901    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764904    2577 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764907    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764910    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764913    2577 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764916    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764919    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764922    2577 flags.go:64] FLAG: --system-cgroups=""
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764925    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764931    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764935    2577 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764938    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764942    2577 flags.go:64] FLAG: --tls-min-version=""
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764945    2577 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 14:55:49.772039 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764948    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764951    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764954    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764957    2577 flags.go:64] FLAG: --v="2"
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764962    2577 flags.go:64] FLAG: --version="false"
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764966    2577 flags.go:64] FLAG: --vmodule=""
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764970    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.764973    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765113    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765118    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765121    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765124    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765127    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765130    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765132    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765135    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765138    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765140    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765143    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765145    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765148    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:49.773090 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765150    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765153    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765156    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765158    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765162    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765165    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765168    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765172    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765174    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765177    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765180    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765182    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765185    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765188    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765190    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765193    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765195    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765198    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765200    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:49.773825 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765203    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765205    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765208    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765211    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765214    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765216    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765219    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765221    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765223    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765226    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765229    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765231    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765234    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765236    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765239    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765241    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765244    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765248    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765251    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765254    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:49.774380 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765258    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765261    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765263    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765266    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765268    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765271    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765273    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765276    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765278    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765281    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765283    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765286    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765288    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765291    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765294    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765296    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765299    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765302    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765305    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:49.775245 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765319    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765321    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765324    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765326    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765329    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765332    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765334    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765337    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765340    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765342    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765346    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765349    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765351    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765355    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765358    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765361    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765363    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765366    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765368    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:49.776146 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765371    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765373    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765376    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765378    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765383    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765386    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765389    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765392    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765395    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765399    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765402    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765405    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765409    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765412    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765415    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:49.776940 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765417    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.765445    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765482    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765487    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765490    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765493    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765495    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765498    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765502    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765505    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765507    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765510    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765515    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765518    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765521    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765523    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:49.777421 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765526    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765528    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765531    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765533    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765536    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765538    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765541    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765544    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765546    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765549    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765551    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765555    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765559    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765562    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765565    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765569    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765573    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765576    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765579    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:49.778067 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765582    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765585    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765587    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765590    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765593    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765596    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765599    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765602    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765604    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765610    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765612    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765615    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765618    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765620    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765623    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765626    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765628    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765631    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765633    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765636    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:49.778877 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765638    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765641    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765644    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765646    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765649    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765651    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765654    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765656    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765659    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765662    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765665    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765667    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765670    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765672    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765675    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765677    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765679    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765683    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765686    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765689    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:49.779386 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765691    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765695    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765698    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765700    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765703    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765706    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765708    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765711    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765714    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765716    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765719    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765721    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765724    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765727    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765729    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765732    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765734    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765737    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765739    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:49.779996 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765742    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765744    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765747    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765750    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765752    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765755    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765757    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765760    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765762    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765765    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765769    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765772    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765775    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.765777    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.773543    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.773560    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.774206    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.774216    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774304    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:49.780510 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774330    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774337    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774342    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774348    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774352    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774357    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774361    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774366    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774370    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774375    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774379    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774384    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774387    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774392    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774395    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774401    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774405    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774410    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774415    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774419    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:49.781019 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774424    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774428    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774432    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774437    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774442    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774446    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774450    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774455    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774459    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774464    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774468    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774473    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774477    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774481    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774486    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774490    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774494    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774499    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774503    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774508    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:49.781551 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774512    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774516    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774521    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774525    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774537    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774542    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774546    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774550    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774558    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774564    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774570    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774574    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774579    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774583    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774587    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774592    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774596    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774601    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774605    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:49.782020 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774609    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774614    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774618    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774623    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774627    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774631    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774636    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774640    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774644    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774649    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774653    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774657    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774662    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774666    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774671    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774675    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774679    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774683    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774688    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774692    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:49.782549 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774696    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774701    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774705    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774710    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774715    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774719    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774723    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774727    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774731    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774736    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774740    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774745    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774750    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774755    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774759    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774764    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774768    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774773    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774777    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774781    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:49.783229 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774786    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774790    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774794    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774801    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774807    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.774812    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.774855    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775078    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775086    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775091    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775097    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775102    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775106    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775110    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775114    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:49.784080 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775119    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775124    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775129    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775134    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775138    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775142    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775146    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775151    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775155    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775159    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775164    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775168    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775172    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775177    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775181    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775186    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775190    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775194    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:49.784734 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775200    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775205    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775210    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775215    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775219    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775223    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775227    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775231    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775235    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775239    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775243    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775247    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775251    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775255    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775259    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775264    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775269    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775275    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775282    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:49.785489 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775287    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775292    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775296    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775301    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775305    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775325    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775329    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775333    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775337    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775341    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775345    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775349    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775352    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775356    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775360    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775365    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775369    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775373    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775377    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775381    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:49.785943 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775385    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775389    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775393    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775397    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775401    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775405    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775409    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775413    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775417    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775422    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775427    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775432    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775436    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775440    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775444    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775448    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775453    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775457    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775461    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:49.786461 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775465    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775470    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775474    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775478    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775482    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775486    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775491    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775495    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775500    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775504    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775508    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775513    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775517    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775521    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775525    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775529    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775534    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775538    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775542    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775546    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:49.786922 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775551    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775555    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.775600    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775827    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775842    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775847    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775852    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775863    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775869    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775874    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775878    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775884    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775888    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775893    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775897    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775903    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:49.787456 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775907    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775912    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775917    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775930    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775938    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775943    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775947    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775954    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775958    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775964    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775968    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775973    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775977    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775981    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775986    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.775991    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776000    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776005    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776010    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776014    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:49.787828 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776052    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776058    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776062    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776066    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776071    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776076    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776080    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776084    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776093    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776098    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776102    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776107    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776112    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776116    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776121    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776125    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776129    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776133    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776138    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:49.788371 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776143    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776147    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776158    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776164    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776169    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776174    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776180    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776185    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776190    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776194    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776199    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776203    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776208    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776212    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776245    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776255    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776259    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776264    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776268    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:49.788824 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776277    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776283    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776288    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776294    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776298    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776303    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776322    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776331    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776341    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776345    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776350    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776354    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776358    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776363    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776368    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776372    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776376    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776381    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776385    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776390    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:49.789264 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776399    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776404    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776408    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776413    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776417    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776421    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776426    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776430    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776464    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776469    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776507    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776513    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776517    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776531    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:49.776536    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.776713    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.779587    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.780612    2577 server.go:1014] "Starting client certificate rotation"
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.780710    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 14:55:49.789741 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.781426    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 14:55:49.790181 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.783512    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:55:49.790181 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.785337    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:55:49.795953 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.795937    2577 log.go:25] "Validated CRI v1 runtime API"
Nov 02 14:55:49.799342 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.799327    2577 log.go:25] "Validated CRI v1 image API"
Nov 02 14:55:49.800291 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.800259    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 14:55:49.803693 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.803665    2577 fs.go:135] Filesystem UUIDs: map[1ac83c1d-9ea1-408d-adec-86cd8565b53d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b2396700-4548-426b-b27a-196eecbb0b4f:/dev/nvme0n1p3]
Nov 02 14:55:49.803742 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.803694    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 14:55:49.808818 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.808701    2577 manager.go:217] Machine: {Timestamp:2026-11-02 14:55:49.807604609 +0000 UTC m=+0.302198525 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099945 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fc4aea2d11a000b4dd42cf0126e38 SystemUUID:ec2fc4ae-a2d1-1a00-0b4d-d42cf0126e38 BootID:603f1e91-e129-4a21-b778-42331279ed2c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:13:1f:9f:0a:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:13:1f:9f:0a:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:af:77:49:6c:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 14:55:49.809360 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.809349    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 14:55:49.809474 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.809461    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 14:55:49.813442 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.813415    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 14:55:49.813587 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.813445    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-27.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 14:55:49.813636 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.813589    2577 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 14:55:49.813636 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.813598    2577 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 14:55:49.813636 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.813634    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:55:49.813711 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.813649    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:55:49.814905 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.814893    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 14:55:49.818999 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.818986    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 14:55:49.819109 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.819100    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 14:55:49.819844 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.819835    2577 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 14:55:49.819883 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.819854    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 14:55:49.819883 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.819866    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 14:55:49.819883 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.819876    2577 kubelet.go:405] "Adding apiserver pod source"
Nov 02 14:55:49.819970 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.819885    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 14:55:49.822865 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.822840    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:55:49.825163 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.825146    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 14:55:49.831485 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.831471    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:55:49.832284 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.832272    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 14:55:49.832347 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.832302    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 14:55:49.832470 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.832460    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 14:55:49.832504 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.832476    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 14:55:49.832504 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.832482    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 14:55:49.832504 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.832487    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 14:55:49.832504 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.832493    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 14:55:49.832504 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.832498    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 14:55:49.833043 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.833034    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 14:55:49.833072 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.833043    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 14:55:49.833072 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.833051    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 14:55:49.833072 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.833057    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 14:55:49.833072 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.833066    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 14:55:49.833226 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.833075    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 14:55:49.833687 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.833678    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 14:55:49.833720 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.833691    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 14:55:49.837548 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.837536    2577 server.go:1263] "Started kubelet"
Nov 02 14:55:49.837654 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.837624    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 14:55:49.837755 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.837712    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 14:55:49.837805 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.837783    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 14:55:49.839467 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.839448    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 14:55:49.839750 ip-10-0-132-27 systemd[1]: Started Kubernetes Kubelet.
Nov 02 14:55:49.841730 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.841716    2577 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 14:55:49.846172 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:49.846155    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 14:55:49.849701 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.849680    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 14:55:49.850420 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.850408    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 14:55:49.850502 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:49.850477    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:49.851107 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.851075    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 14:55:49.851107 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.851075    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 14:55:49.851107 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.851099    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 14:55:49.851304 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.851178    2577 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 14:55:49.851304 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.851186    2577 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 14:55:49.851570 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.851554    2577 factory.go:153] Registering CRI-O factory
Nov 02 14:55:49.851629 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.851575    2577 factory.go:223] Registration of the crio container factory successfully
Nov 02 14:55:49.851681 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.851669    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 14:55:49.851715 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.851683    2577 factory.go:55] Registering systemd factory
Nov 02 14:55:49.851715 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.851694    2577 factory.go:223] Registration of the systemd container factory successfully
Nov 02 14:55:49.851715 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.851711    2577 factory.go:103] Registering Raw factory
Nov 02 14:55:49.851860 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.851724    2577 manager.go:1199] Started watching for new ooms in manager
Nov 02 14:55:49.852766 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.852692    2577 manager.go:319] Starting recovery of all containers
Nov 02 14:55:49.867137 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.867119    2577 manager.go:324] Recovery completed
Nov 02 14:55:49.871286 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.871270    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:49.872940 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.872925    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:49.873016 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.872952    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:49.873016 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.872963    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:55:49.873540 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.873523    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:49.873632 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.873521    2577 cpu_manager.go:226] "Starting" policy="none"
Nov 02 14:55:49.873632 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.873555    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 14:55:49.873632 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.873574    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 14:55:49.874193 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.874175    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:49.874268 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.874210    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:49.874268 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.874224    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:55:49.875781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.875768    2577 policy_none.go:50] "Start"
Nov 02 14:55:49.875830 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.875786    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 14:55:49.875830 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.875799    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 14:55:49.877753 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.877741    2577 policy_none.go:44] "Start"
Nov 02 14:55:49.908947 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.908909    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 14:55:49.911535 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.911520    2577 manager.go:341] "Starting Device Plugin manager"
Nov 02 14:55:49.911629 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:49.911552    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 14:55:49.911629 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.911563    2577 server.go:87] "Starting device plugin registration server"
Nov 02 14:55:49.911804 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.911791    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 14:55:49.911864 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.911807    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 14:55:49.911909 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.911880    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 14:55:49.912368 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.912245    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 14:55:49.912368 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.912259    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 14:55:49.912645 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:49.912627    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 14:55:49.912686 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:49.912678    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:49.965159 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.965133    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 14:55:49.966994 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.966385    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 14:55:49.966994 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.966406    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 14:55:49.966994 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.966429    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 14:55:49.966994 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:49.966463    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 14:55:49.982077 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:49.982046    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-27.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 14:55:49.982077 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:49.982062    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-27.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 14:55:49.982984 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:49.982058    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-27.us-west-1.compute.internal.18e44235a911b0dc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-27.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-27.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 14:55:49.837508828 +0000 UTC m=+0.332102741,LastTimestamp:2026-11-02 14:55:49.837508828 +0000 UTC m=+0.332102741,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-27.us-west-1.compute.internal,}"
Nov 02 14:55:50.012552 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.012498    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:50.013302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.013285    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:50.013406 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.013334    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:50.013406 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.013351    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:55:50.013406 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.013395    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.037924 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.037880    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 14:55:50.037924 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.037841    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-27.us-west-1.compute.internal.18e44235ab2e50e1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-27.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-27.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-27.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 14:55:49.872939233 +0000 UTC m=+0.367533146,LastTimestamp:2026-11-02 14:55:49.872939233 +0000 UTC m=+0.367533146,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-27.us-west-1.compute.internal,}"
Nov 02 14:55:50.038114 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.037960    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-27.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 14:55:50.060077 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.060056    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 14:55:50.060161 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.060111    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9pr52"
Nov 02 14:55:50.067094 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.067072    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"]
Nov 02 14:55:50.067178 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.067125    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:50.067999 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.067984    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:50.068072 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.068014    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:50.068072 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.068027    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:55:50.070155 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.070145    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:50.070293 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.070280    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.070366 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.070331    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:50.070868 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.070853    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:50.070931 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.070869    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:50.070931 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.070881    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:50.070931 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.070889    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:50.070931 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.070903    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:55:50.071107 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.070891    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:55:50.073093 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.073077    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.073172 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.073104    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:50.073746 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.073733    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:50.073816 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.073759    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:50.073816 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.073773    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:55:50.106633 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.106617    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found" node="ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.109361 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.109345    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found" node="ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.140566 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.140548    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:50.140566 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.140556    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:50.148922 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.148908    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.148994 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.148934    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-27.us-west-1.compute.internal\": node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:50.252012 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.251978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/632223b57cdaa2366b6d306d2b0b8330-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"632223b57cdaa2366b6d306d2b0b8330\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.252167 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.252034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/632223b57cdaa2366b6d306d2b0b8330-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"632223b57cdaa2366b6d306d2b0b8330\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.252167 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.252052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c52ee3e32e488dfc59e1227cdb8da56d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"c52ee3e32e488dfc59e1227cdb8da56d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.258534 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.258506    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:50.352420 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.352335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/632223b57cdaa2366b6d306d2b0b8330-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"632223b57cdaa2366b6d306d2b0b8330\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.352420 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.352368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/632223b57cdaa2366b6d306d2b0b8330-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"632223b57cdaa2366b6d306d2b0b8330\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.352420 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.352387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c52ee3e32e488dfc59e1227cdb8da56d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"c52ee3e32e488dfc59e1227cdb8da56d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.352642 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.352438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c52ee3e32e488dfc59e1227cdb8da56d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"c52ee3e32e488dfc59e1227cdb8da56d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.352642 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.352464    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/632223b57cdaa2366b6d306d2b0b8330-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"632223b57cdaa2366b6d306d2b0b8330\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.352642 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.352466    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/632223b57cdaa2366b6d306d2b0b8330-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"632223b57cdaa2366b6d306d2b0b8330\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.359423 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.359404    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:50.411236 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.411212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.413361 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.413342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:50.459704 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.459681    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:50.560200 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.560170    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:50.660761 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.660697    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:50.761178 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.761154    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:50.780073 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.780047    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 14:55:50.780223 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.780144    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 14:55:50.780223 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:50.780157    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 14:55:50.862120 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.862085    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:50.963422 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:50.963163    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:51.050485 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.050457    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:51.051608 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.051587    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:51.051735 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.051619    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:51.051735 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.051630    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:55:51.063731 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:51.063693    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:51.066042 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.066004    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:50:50 +0000 UTC" deadline="2028-05-27 05:53:15.950646576 +0000 UTC"
Nov 02 14:55:51.066090 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.066043    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13718h57m24.884606936s"
Nov 02 14:55:51.163987 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:51.163952    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:51.264421 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:51.264334    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:55:51.288676 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.288656    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:51.351489 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.351459    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:51.421249 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.421226    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:55:51.423156 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.423142    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:55:51.488827 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.488805    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:55:51.676882 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:51.676853    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52ee3e32e488dfc59e1227cdb8da56d.slice/crio-9a07090f931f3077f7a532bbdd8e28e8c071f4f7870bcecd391fabb7ea9cdbc0 WatchSource:0}: Error finding container 9a07090f931f3077f7a532bbdd8e28e8c071f4f7870bcecd391fabb7ea9cdbc0: Status 404 returned error can't find the container with id 9a07090f931f3077f7a532bbdd8e28e8c071f4f7870bcecd391fabb7ea9cdbc0
Nov 02 14:55:51.679001 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:51.678974    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632223b57cdaa2366b6d306d2b0b8330.slice/crio-365b3df95f728f6953f143442eb4f2a78449283c2a46402d6fac2c87491832dc WatchSource:0}: Error finding container 365b3df95f728f6953f143442eb4f2a78449283c2a46402d6fac2c87491832dc: Status 404 returned error can't find the container with id 365b3df95f728f6953f143442eb4f2a78449283c2a46402d6fac2c87491832dc
Nov 02 14:55:51.684329 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.684312    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:55:51.696238 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.696221    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:51.821133 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.821061    2577 apiserver.go:52] "Watching apiserver"
Nov 02 14:55:51.850813 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.850785    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 14:55:51.869843 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.869818    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:51.931731 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.931704    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 14:55:51.932049 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.932028    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-jjhkj","kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-5lhfv","openshift-image-registry/node-ca-9htnf","openshift-network-operator/iptables-alerter-whl44","openshift-ovn-kubernetes/ovnkube-node-8822k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-596b4","openshift-multus/multus-w5w2m","kube-system/konnectivity-agent-spbjm","openshift-dns/node-resolver-cllms","kube-system/global-pull-secret-syncer-nx6px","openshift-multus/network-metrics-daemon-rkxt7"]
Nov 02 14:55:51.936157 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.936141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.938372 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.938348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:55:51.940773 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.940753    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:51.940855 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:51.940836    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:55:51.943170 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.943152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:55:51.946867 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.946843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:55:51.949496 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.949477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.951885 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.951870    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:51.954348 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.954326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:55:51.954447 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:51.954409    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:55:51.956810 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.956787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:51.956911 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:51.956864    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:55:51.959330 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.959303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:51.961033 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:51.961150 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n9ql\" (UniqueName: \"kubernetes.io/projected/acede18e-4d8e-47ee-b31f-918658a9ffba-kube-api-access-7n9ql\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:55:51.961150 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:51.961150 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961074    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-run-netns\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.961150 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.961390 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovnkube-script-lib\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.961390 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-daemon-config\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.961390 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cnibin\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:51.961390 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-kubelet\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.961390 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-cni-bin\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.961390 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-socket-dir-parent\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.961390 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npd9z\" (UniqueName: \"kubernetes.io/projected/100e3924-bcfc-4208-84ea-c1ff1656e949-kube-api-access-npd9z\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.961745 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx97c\" (UniqueName: \"kubernetes.io/projected/9bea213e-5ff8-447d-98db-fa0ec51cda10-kube-api-access-bx97c\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:51.961745 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-systemd-units\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.961745 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-node-log\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.961745 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-system-cni-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.961745 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-cni-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.961745 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-k8s-cni-cncf-io\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.961745 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-cni-bin\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.961745 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-kubelet\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.961745 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-var-lib-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.961745 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.961745 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acede18e-4d8e-47ee-b31f-918658a9ffba-serviceca\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-system-cni-dir\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7d93f72-6fc0-469a-8c16-9feed09bda86-konnectivity-ca\") pod \"konnectivity-agent-spbjm\" (UID: \"c7d93f72-6fc0-469a-8c16-9feed09bda86\") " pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bb718bc-429a-4875-bd8d-d52dbba0b252-iptables-alerter-script\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bb718bc-429a-4875-bd8d-d52dbba0b252-host-slash\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-hostroot\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-cnibin\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-netns\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.961983    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cllms"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-conf-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-etc-kubernetes\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-ovn\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-log-socket\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-cni-netd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-cni-multus\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-multus-certs\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.962331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cni-binary-copy\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8n6k\" (UniqueName: \"kubernetes.io/projected/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-kube-api-access-b8n6k\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-slash\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-run-ovn-kubernetes\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovn-node-metrics-cert\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmfrd\" (UniqueName: \"kubernetes.io/projected/6f2ebec5-9906-438f-b2ce-010345fa49ba-kube-api-access-rmfrd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/100e3924-bcfc-4208-84ea-c1ff1656e949-cni-binary-copy\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-etc-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acede18e-4d8e-47ee-b31f-918658a9ffba-host\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7d93f72-6fc0-469a-8c16-9feed09bda86-agent-certs\") pod \"konnectivity-agent-spbjm\" (UID: \"c7d93f72-6fc0-469a-8c16-9feed09bda86\") " pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzqh5\" (UniqueName: \"kubernetes.io/projected/9bb718bc-429a-4875-bd8d-d52dbba0b252-kube-api-access-vzqh5\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-systemd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovnkube-config\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.963235 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.962754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-env-overrides\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:51.965684 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.965664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:51.970727 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.970686    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal" event={"ID":"632223b57cdaa2366b6d306d2b0b8330","Type":"ContainerStarted","Data":"365b3df95f728f6953f143442eb4f2a78449283c2a46402d6fac2c87491832dc"}
Nov 02 14:55:51.971824 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:51.971797    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal" event={"ID":"c52ee3e32e488dfc59e1227cdb8da56d","Type":"ContainerStarted","Data":"9a07090f931f3077f7a532bbdd8e28e8c071f4f7870bcecd391fabb7ea9cdbc0"}
Nov 02 14:55:52.000573 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.000545    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 14:55:52.000673 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.000574    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5w9z9\""
Nov 02 14:55:52.000673 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.000549    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 14:55:52.000673 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.000574    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 14:55:52.000673 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.000632    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 14:55:52.003096 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.003079    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 14:55:52.043591 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.043561    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 14:55:52.043700 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.043628    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kb4st"
Nov 02 14:55:52.052788 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.052770    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 14:55:52.062914 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.062889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-log-socket\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.062961 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.062919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-cni-netd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.062961 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.062938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovn-node-metrics-cert\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.062961 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.062954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/100e3924-bcfc-4208-84ea-c1ff1656e949-cni-binary-copy\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.063072 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063013    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-cni-netd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.063072 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063039    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-log-socket\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.063072 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2pbg\" (UniqueName: \"kubernetes.io/projected/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-kube-api-access-k2pbg\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.063177 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-modprobe-d\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.063177 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-kubernetes\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.063243 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-lib-modules\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.063243 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-var-lib-kubelet\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.063243 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-etc-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.063378 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acede18e-4d8e-47ee-b31f-918658a9ffba-host\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:55:52.063378 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063270    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-etc-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.063378 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063294    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acede18e-4d8e-47ee-b31f-918658a9ffba-host\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:55:52.063378 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzqh5\" (UniqueName: \"kubernetes.io/projected/9bb718bc-429a-4875-bd8d-d52dbba0b252-kube-api-access-vzqh5\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:55:52.063378 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-systemd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.063378 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovnkube-script-lib\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.063625 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-kubelet-config\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:52.063625 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.063625 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysconfig\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.063625 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063471    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-systemd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.063625 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.063625 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063497    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/100e3924-bcfc-4208-84ea-c1ff1656e949-cni-binary-copy\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.063625 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.063625 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-daemon-config\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.063625 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:52.063625 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-systemd\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.063625 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cnibin\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063635    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-cni-bin\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063722    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063750    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cnibin\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-socket-dir-parent\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063816    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-cni-bin\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-socket-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-device-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx97c\" (UniqueName: \"kubernetes.io/projected/9bea213e-5ff8-447d-98db-fa0ec51cda10-kube-api-access-bx97c\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-system-cni-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063969    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-socket-dir-parent\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-k8s-cni-cncf-io\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.063983    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-daemon-config\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064016    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-k8s-cni-cncf-io\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064058 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-sys\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-tuned\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bb718bc-429a-4875-bd8d-d52dbba0b252-host-slash\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064247    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-system-cni-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-hostroot\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064291    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bb718bc-429a-4875-bd8d-d52dbba0b252-host-slash\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-conf-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064327    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-hostroot\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-netns\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064423    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-conf-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-sys-fs\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-netns\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-ovn\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-cni-multus\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064517    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-ovn\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-multus-certs\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74049a08-892a-4ae2-a5a5-161da6182c1c-tmp-dir\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:55:52.064647 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064619    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-multus-certs\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064617    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-cni-multus\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96w6k\" (UniqueName: \"kubernetes.io/projected/74049a08-892a-4ae2-a5a5-161da6182c1c-kube-api-access-96w6k\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cni-binary-copy\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8n6k\" (UniqueName: \"kubernetes.io/projected/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-kube-api-access-b8n6k\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-slash\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-run-ovn-kubernetes\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmfrd\" (UniqueName: \"kubernetes.io/projected/6f2ebec5-9906-438f-b2ce-010345fa49ba-kube-api-access-rmfrd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064833    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-slash\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064833    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-run-ovn-kubernetes\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7d93f72-6fc0-469a-8c16-9feed09bda86-agent-certs\") pod \"konnectivity-agent-spbjm\" (UID: \"c7d93f72-6fc0-469a-8c16-9feed09bda86\") " pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.064966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovnkube-config\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-env-overrides\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ce39d1-e591-4af6-81e4-63d7d7b255c4-tmp\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsk26\" (UniqueName: \"kubernetes.io/projected/19ce39d1-e591-4af6-81e4-63d7d7b255c4-kube-api-access-dsk26\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.065436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9ql\" (UniqueName: \"kubernetes.io/projected/acede18e-4d8e-47ee-b31f-918658a9ffba-kube-api-access-7n9ql\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-run-netns\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-dbus\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysctl-d\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-kubelet\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065271    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cni-binary-copy\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npd9z\" (UniqueName: \"kubernetes.io/projected/100e3924-bcfc-4208-84ea-c1ff1656e949-kube-api-access-npd9z\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.065306    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065332    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-kubelet\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-registration-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysctl-conf\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-host\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.065439    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs podName:9bea213e-5ff8-447d-98db-fa0ec51cda10 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.565406535 +0000 UTC m=+3.060000473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs") pod "network-metrics-daemon-rkxt7" (UID: "9bea213e-5ff8-447d-98db-fa0ec51cda10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-run-netns\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-systemd-units\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065515    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-systemd-units\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.066102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-node-log\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-cni-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065560    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-node-log\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-cni-bin\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065591    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-cni-bin\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-kubelet\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-run\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065664    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-kubelet\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-var-lib-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065673    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-cni-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acede18e-4d8e-47ee-b31f-918658a9ffba-serviceca\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-system-cni-dir\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065735    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7d93f72-6fc0-469a-8c16-9feed09bda86-konnectivity-ca\") pod \"konnectivity-agent-spbjm\" (UID: \"c7d93f72-6fc0-469a-8c16-9feed09bda86\") " pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065700    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-var-lib-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bb718bc-429a-4875-bd8d-d52dbba0b252-iptables-alerter-script\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065788    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-system-cni-dir\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.066619 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-cnibin\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.067129 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-etc-kubernetes\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.067129 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74049a08-892a-4ae2-a5a5-161da6182c1c-hosts-file\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:55:52.067129 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065875    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-cnibin\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.067129 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw78t\" (UniqueName: \"kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t\") pod \"network-check-target-jjhkj\" (UID: \"8eaf7fe9-c300-4798-8212-e57e5a9cb03d\") " pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:55:52.067129 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.065911    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-etc-kubernetes\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.067129 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.066121    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7d93f72-6fc0-469a-8c16-9feed09bda86-konnectivity-ca\") pod \"konnectivity-agent-spbjm\" (UID: \"c7d93f72-6fc0-469a-8c16-9feed09bda86\") " pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:55:52.108860 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.108682    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 14:55:52.108860 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.108689    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 14:55:52.108860 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.108716    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 14:55:52.108860 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.108734    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:52.108860 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.108779    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 14:55:52.108860 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.108789    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 14:55:52.108860 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.108791    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 14:55:52.109395 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.108869    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 14:55:52.109395 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.108897    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 14:55:52.109395 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109088    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 14:55:52.109395 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109128    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 14:55:52.109395 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109132    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 14:55:52.109395 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109166    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:52.109395 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109167    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 14:55:52.109395 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109185    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:52.109395 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109177    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zp9vx\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109491    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5lvgl\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109504    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109520    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8dn6m\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109528    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109542    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-snsjj\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109553    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zct6j\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109539    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109542    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109555    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109572    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109600    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m4lkk\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109610    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dhf26\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109629    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 14:55:52.109781 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.109633    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rrdxh\""
Nov 02 14:55:52.114325 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.114295    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 14:55:52.114564 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.114547    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovnkube-script-lib\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.114926 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.114907    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.115315 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.115300    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.116563 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.116536    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-env-overrides\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.116694 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.116600    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovnkube-config\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.116841 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.116821    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bb718bc-429a-4875-bd8d-d52dbba0b252-iptables-alerter-script\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:55:52.116841 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.116831    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acede18e-4d8e-47ee-b31f-918658a9ffba-serviceca\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:55:52.117896 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.117877    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovn-node-metrics-cert\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.118405 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.118369    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7d93f72-6fc0-469a-8c16-9feed09bda86-agent-certs\") pod \"konnectivity-agent-spbjm\" (UID: \"c7d93f72-6fc0-469a-8c16-9feed09bda86\") " pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:55:52.130244 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.130216    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzqh5\" (UniqueName: \"kubernetes.io/projected/9bb718bc-429a-4875-bd8d-d52dbba0b252-kube-api-access-vzqh5\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:55:52.130364 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.130266    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8n6k\" (UniqueName: \"kubernetes.io/projected/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-kube-api-access-b8n6k\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.130682 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.130660    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9ql\" (UniqueName: \"kubernetes.io/projected/acede18e-4d8e-47ee-b31f-918658a9ffba-kube-api-access-7n9ql\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:55:52.130682 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.130672    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmfrd\" (UniqueName: \"kubernetes.io/projected/6f2ebec5-9906-438f-b2ce-010345fa49ba-kube-api-access-rmfrd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.130798 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.130726    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npd9z\" (UniqueName: \"kubernetes.io/projected/100e3924-bcfc-4208-84ea-c1ff1656e949-kube-api-access-npd9z\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.130959 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.130943    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx97c\" (UniqueName: \"kubernetes.io/projected/9bea213e-5ff8-447d-98db-fa0ec51cda10-kube-api-access-bx97c\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:52.166302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-sys-fs\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.166382 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74049a08-892a-4ae2-a5a5-161da6182c1c-tmp-dir\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:55:52.166382 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96w6k\" (UniqueName: \"kubernetes.io/projected/74049a08-892a-4ae2-a5a5-161da6182c1c-kube-api-access-96w6k\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:55:52.166382 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.166503 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ce39d1-e591-4af6-81e4-63d7d7b255c4-tmp\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.166503 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsk26\" (UniqueName: \"kubernetes.io/projected/19ce39d1-e591-4af6-81e4-63d7d7b255c4-kube-api-access-dsk26\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.166503 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166487    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-sys-fs\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.166636 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166545    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.166636 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-dbus\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:52.166636 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysctl-d\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.166636 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-registration-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysctl-conf\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-host\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166700    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-registration-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166647    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-dbus\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-run\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166724    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74049a08-892a-4ae2-a5a5-161da6182c1c-tmp-dir\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166744    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysctl-d\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74049a08-892a-4ae2-a5a5-161da6182c1c-hosts-file\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166763    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysctl-conf\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166776    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-run\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw78t\" (UniqueName: \"kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t\") pod \"network-check-target-jjhkj\" (UID: \"8eaf7fe9-c300-4798-8212-e57e5a9cb03d\") " pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166764    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-host\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166805    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74049a08-892a-4ae2-a5a5-161da6182c1c-hosts-file\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:55:52.166822 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2pbg\" (UniqueName: \"kubernetes.io/projected/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-kube-api-access-k2pbg\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-modprobe-d\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-kubernetes\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-lib-modules\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-var-lib-kubelet\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-kubelet-config\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166952    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-kubernetes\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166952    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-modprobe-d\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.166996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysconfig\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167006    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-lib-modules\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167015    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-kubelet-config\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167007    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-var-lib-kubelet\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167009    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167060    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysconfig\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-systemd\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.167103    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:52.167484 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167119    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-systemd\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.168005 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-socket-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.168005 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.167162    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret podName:41cff577-8a79-4b9e-89c8-a93e1ecf4ff0 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.667148662 +0000 UTC m=+3.161742567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret") pod "global-pull-secret-syncer-nx6px" (UID: "41cff577-8a79-4b9e-89c8-a93e1ecf4ff0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:52.168005 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-device-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.168005 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-sys\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.168005 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-tuned\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.168005 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167223    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-socket-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.168005 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167264    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-device-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.168005 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.167276    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-sys\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.168579 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.168557    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ce39d1-e591-4af6-81e4-63d7d7b255c4-tmp\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.169214 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.169194    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-tuned\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.230084 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.230059    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:52.230084 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.230080    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:52.230084 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.230092    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hw78t for pod openshift-network-diagnostics/network-check-target-jjhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:52.230291 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.230145    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t podName:8eaf7fe9-c300-4798-8212-e57e5a9cb03d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.730131235 +0000 UTC m=+3.224725137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hw78t" (UniqueName: "kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t") pod "network-check-target-jjhkj" (UID: "8eaf7fe9-c300-4798-8212-e57e5a9cb03d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:52.232314 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.232293    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2pbg\" (UniqueName: \"kubernetes.io/projected/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-kube-api-access-k2pbg\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.232570 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.232547    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96w6k\" (UniqueName: \"kubernetes.io/projected/74049a08-892a-4ae2-a5a5-161da6182c1c-kube-api-access-96w6k\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:55:52.232617 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.232549    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsk26\" (UniqueName: \"kubernetes.io/projected/19ce39d1-e591-4af6-81e4-63d7d7b255c4-kube-api-access-dsk26\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.248758 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.248743    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w5w2m"
Nov 02 14:55:52.256463 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.256445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:55:52.259238 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:52.259213    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100e3924_bcfc_4208_84ea_c1ff1656e949.slice/crio-e8d79991e743cf3435c16fdb7859603a3e1eec5bad724f20f1ea2a923cd7e6ba WatchSource:0}: Error finding container e8d79991e743cf3435c16fdb7859603a3e1eec5bad724f20f1ea2a923cd7e6ba: Status 404 returned error can't find the container with id e8d79991e743cf3435c16fdb7859603a3e1eec5bad724f20f1ea2a923cd7e6ba
Nov 02 14:55:52.263706 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.263686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:55:52.265234 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:52.265203    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d93f72_6fc0_469a_8c16_9feed09bda86.slice/crio-61642b8959214064dd84b6bb614a5e9d97a10e840e2dd10b2115977f7d660a2c WatchSource:0}: Error finding container 61642b8959214064dd84b6bb614a5e9d97a10e840e2dd10b2115977f7d660a2c: Status 404 returned error can't find the container with id 61642b8959214064dd84b6bb614a5e9d97a10e840e2dd10b2115977f7d660a2c
Nov 02 14:55:52.272410 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.272375    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:52.272891 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:52.272870    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacede18e_4d8e_47ee_b31f_918658a9ffba.slice/crio-0d45f714830b537a2feb50896809b1fe72666081846d09c5931670e07a683698 WatchSource:0}: Error finding container 0d45f714830b537a2feb50896809b1fe72666081846d09c5931670e07a683698: Status 404 returned error can't find the container with id 0d45f714830b537a2feb50896809b1fe72666081846d09c5931670e07a683698
Nov 02 14:55:52.281371 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.281349    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:55:52.286326 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.286305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:55:52.290979 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:52.290959    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb718bc_429a_4875_bd8d_d52dbba0b252.slice/crio-09f5f8c60b67e971cecefc180ad34912a7c688b3b0d7a056d99b90caabd7703b WatchSource:0}: Error finding container 09f5f8c60b67e971cecefc180ad34912a7c688b3b0d7a056d99b90caabd7703b: Status 404 returned error can't find the container with id 09f5f8c60b67e971cecefc180ad34912a7c688b3b0d7a056d99b90caabd7703b
Nov 02 14:55:52.296103 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:52.296081    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e22bc1a_6a91_4b05_94ce_e3f11ccb0eac.slice/crio-577421c14bf19c27ebec20ab6b2456e2f9f8b28f8e848db135575c07900c7cf8 WatchSource:0}: Error finding container 577421c14bf19c27ebec20ab6b2456e2f9f8b28f8e848db135575c07900c7cf8: Status 404 returned error can't find the container with id 577421c14bf19c27ebec20ab6b2456e2f9f8b28f8e848db135575c07900c7cf8
Nov 02 14:55:52.296696 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.296681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:55:52.299348 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.299331    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cllms"
Nov 02 14:55:52.303794 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.303775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:55:52.306231 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:52.306207    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e40d7f_ab6a_4817_ba3e_3e2ab361b454.slice/crio-451041e2651890f10aa467d5f33ade18f569e0bf4afcf49f25fe2f64da05ea15 WatchSource:0}: Error finding container 451041e2651890f10aa467d5f33ade18f569e0bf4afcf49f25fe2f64da05ea15: Status 404 returned error can't find the container with id 451041e2651890f10aa467d5f33ade18f569e0bf4afcf49f25fe2f64da05ea15
Nov 02 14:55:52.309855 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:52.309833    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74049a08_892a_4ae2_a5a5_161da6182c1c.slice/crio-dc884cd696d8fd330a276edb5d95785996d300ff3d789a0024c43253e48a12e4 WatchSource:0}: Error finding container dc884cd696d8fd330a276edb5d95785996d300ff3d789a0024c43253e48a12e4: Status 404 returned error can't find the container with id dc884cd696d8fd330a276edb5d95785996d300ff3d789a0024c43253e48a12e4
Nov 02 14:55:52.312038 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.312008    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:55:52.315486 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:52.315463    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ce39d1_e591_4af6_81e4_63d7d7b255c4.slice/crio-2a8ce396b50c2290bcf546a8a6afba1da52cc9710bdfc2e49a8f722a7f8700fe WatchSource:0}: Error finding container 2a8ce396b50c2290bcf546a8a6afba1da52cc9710bdfc2e49a8f722a7f8700fe: Status 404 returned error can't find the container with id 2a8ce396b50c2290bcf546a8a6afba1da52cc9710bdfc2e49a8f722a7f8700fe
Nov 02 14:55:52.321496 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:55:52.321477    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2ebec5_9906_438f_b2ce_010345fa49ba.slice/crio-b993e107d2170bf2292fd26237cb9e2465fdef5970b188455b859a69283e5ece WatchSource:0}: Error finding container b993e107d2170bf2292fd26237cb9e2465fdef5970b188455b859a69283e5ece: Status 404 returned error can't find the container with id b993e107d2170bf2292fd26237cb9e2465fdef5970b188455b859a69283e5ece
Nov 02 14:55:52.569038 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.568957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:52.569179 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.569059    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:52.569179 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.569122    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs podName:9bea213e-5ff8-447d-98db-fa0ec51cda10 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:53.56910889 +0000 UTC m=+4.063702792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs") pod "network-metrics-daemon-rkxt7" (UID: "9bea213e-5ff8-447d-98db-fa0ec51cda10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:52.669840 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.669802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:52.670017 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.669935    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:52.670017 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.669994    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret podName:41cff577-8a79-4b9e-89c8-a93e1ecf4ff0 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:53.669977711 +0000 UTC m=+4.164571612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret") pod "global-pull-secret-syncer-nx6px" (UID: "41cff577-8a79-4b9e-89c8-a93e1ecf4ff0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:52.770722 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.770680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw78t\" (UniqueName: \"kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t\") pod \"network-check-target-jjhkj\" (UID: \"8eaf7fe9-c300-4798-8212-e57e5a9cb03d\") " pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:55:52.770945 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.770908    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:52.770945 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.770941    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:52.771090 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.770953    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hw78t for pod openshift-network-diagnostics/network-check-target-jjhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:52.771090 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:52.771041    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t podName:8eaf7fe9-c300-4798-8212-e57e5a9cb03d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:53.771012794 +0000 UTC m=+4.265606697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hw78t" (UniqueName: "kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t") pod "network-check-target-jjhkj" (UID: "8eaf7fe9-c300-4798-8212-e57e5a9cb03d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:52.975661 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.975575    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t" event={"ID":"49e40d7f-ab6a-4817-ba3e-3e2ab361b454","Type":"ContainerStarted","Data":"451041e2651890f10aa467d5f33ade18f569e0bf4afcf49f25fe2f64da05ea15"}
Nov 02 14:55:52.976989 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.976944    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-whl44" event={"ID":"9bb718bc-429a-4875-bd8d-d52dbba0b252","Type":"ContainerStarted","Data":"09f5f8c60b67e971cecefc180ad34912a7c688b3b0d7a056d99b90caabd7703b"}
Nov 02 14:55:52.978828 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.978800    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-spbjm" event={"ID":"c7d93f72-6fc0-469a-8c16-9feed09bda86","Type":"ContainerStarted","Data":"61642b8959214064dd84b6bb614a5e9d97a10e840e2dd10b2115977f7d660a2c"}
Nov 02 14:55:52.980554 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.980528    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w5w2m" event={"ID":"100e3924-bcfc-4208-84ea-c1ff1656e949","Type":"ContainerStarted","Data":"e8d79991e743cf3435c16fdb7859603a3e1eec5bad724f20f1ea2a923cd7e6ba"}
Nov 02 14:55:52.982160 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.982123    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8822k" event={"ID":"6f2ebec5-9906-438f-b2ce-010345fa49ba","Type":"ContainerStarted","Data":"b993e107d2170bf2292fd26237cb9e2465fdef5970b188455b859a69283e5ece"}
Nov 02 14:55:52.983687 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.983659    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5lhfv" event={"ID":"19ce39d1-e591-4af6-81e4-63d7d7b255c4","Type":"ContainerStarted","Data":"2a8ce396b50c2290bcf546a8a6afba1da52cc9710bdfc2e49a8f722a7f8700fe"}
Nov 02 14:55:52.985037 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.985011    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cllms" event={"ID":"74049a08-892a-4ae2-a5a5-161da6182c1c","Type":"ContainerStarted","Data":"dc884cd696d8fd330a276edb5d95785996d300ff3d789a0024c43253e48a12e4"}
Nov 02 14:55:52.987372 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.987348    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-596b4" event={"ID":"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac","Type":"ContainerStarted","Data":"577421c14bf19c27ebec20ab6b2456e2f9f8b28f8e848db135575c07900c7cf8"}
Nov 02 14:55:52.988911 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:52.988885    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9htnf" event={"ID":"acede18e-4d8e-47ee-b31f-918658a9ffba","Type":"ContainerStarted","Data":"0d45f714830b537a2feb50896809b1fe72666081846d09c5931670e07a683698"}
Nov 02 14:55:53.047326 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:53.047266    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:52 +0000 UTC" deadline="2028-05-15 08:03:03.065522271 +0000 UTC"
Nov 02 14:55:53.047326 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:53.047332    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13433h7m10.018208785s"
Nov 02 14:55:53.575889 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:53.575847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:53.576096 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:53.576031    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:53.576163 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:53.576111    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs podName:9bea213e-5ff8-447d-98db-fa0ec51cda10 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:55.576090608 +0000 UTC m=+6.070684515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs") pod "network-metrics-daemon-rkxt7" (UID: "9bea213e-5ff8-447d-98db-fa0ec51cda10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:53.676649 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:53.676610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:53.676835 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:53.676778    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:53.676915 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:53.676843    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret podName:41cff577-8a79-4b9e-89c8-a93e1ecf4ff0 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:55.676823974 +0000 UTC m=+6.171417880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret") pod "global-pull-secret-syncer-nx6px" (UID: "41cff577-8a79-4b9e-89c8-a93e1ecf4ff0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:53.777164 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:53.777128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw78t\" (UniqueName: \"kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t\") pod \"network-check-target-jjhkj\" (UID: \"8eaf7fe9-c300-4798-8212-e57e5a9cb03d\") " pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:55:53.777359 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:53.777277    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:53.777359 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:53.777292    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:53.777359 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:53.777301    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hw78t for pod openshift-network-diagnostics/network-check-target-jjhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:53.777552 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:53.777376    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t podName:8eaf7fe9-c300-4798-8212-e57e5a9cb03d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:55.777335257 +0000 UTC m=+6.271929160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hw78t" (UniqueName: "kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t") pod "network-check-target-jjhkj" (UID: "8eaf7fe9-c300-4798-8212-e57e5a9cb03d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:53.970510 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:53.970436    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:55:53.970672 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:53.970577    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:55:53.971048 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:53.971026    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:53.971210 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:53.971189    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:55:53.971311 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:53.971275    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:53.971411 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:53.971349    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:55:54.047903 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:54.047861    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:52 +0000 UTC" deadline="2028-07-20 11:38:35.067896788 +0000 UTC"
Nov 02 14:55:54.047903 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:54.047901    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15020h42m41.02000056s"
Nov 02 14:55:55.593878 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:55.593831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:55.594553 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:55.593992    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:55.594553 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:55.594062    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs podName:9bea213e-5ff8-447d-98db-fa0ec51cda10 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:59.594042818 +0000 UTC m=+10.088636736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs") pod "network-metrics-daemon-rkxt7" (UID: "9bea213e-5ff8-447d-98db-fa0ec51cda10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:55.694870 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:55.694817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:55.695058 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:55.694984    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:55.695126 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:55.695058    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret podName:41cff577-8a79-4b9e-89c8-a93e1ecf4ff0 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:59.695037734 +0000 UTC m=+10.189631650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret") pod "global-pull-secret-syncer-nx6px" (UID: "41cff577-8a79-4b9e-89c8-a93e1ecf4ff0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:55.798055 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:55.797685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw78t\" (UniqueName: \"kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t\") pod \"network-check-target-jjhkj\" (UID: \"8eaf7fe9-c300-4798-8212-e57e5a9cb03d\") " pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:55:55.798055 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:55.797883    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:55.798055 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:55.797905    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:55.798055 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:55.797917    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hw78t for pod openshift-network-diagnostics/network-check-target-jjhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:55.798055 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:55.798000    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t podName:8eaf7fe9-c300-4798-8212-e57e5a9cb03d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:59.797961238 +0000 UTC m=+10.292555155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hw78t" (UniqueName: "kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t") pod "network-check-target-jjhkj" (UID: "8eaf7fe9-c300-4798-8212-e57e5a9cb03d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:55.969684 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:55.969654    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:55.969838 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:55.969811    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:55:55.970067 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:55.970042    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:55.970189 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:55.970168    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:55:55.970269 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:55.970249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:55:55.970405 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:55.970362    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:55:57.013750 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:57.012990    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal" event={"ID":"c52ee3e32e488dfc59e1227cdb8da56d","Type":"ContainerStarted","Data":"c9850c15758c890712e163c210ca46b74a0f8c176c542b704a4c552b62c6d05d"}
Nov 02 14:55:57.024463 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:57.023681    2577 generic.go:356] "Generic (PLEG): container finished" podID="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" containerID="93ab6e9ef87f647231a4772af058c277884e6d6e860f27bb25cb9937738a7d94" exitCode=0
Nov 02 14:55:57.024463 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:57.023769    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-596b4" event={"ID":"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac","Type":"ContainerDied","Data":"93ab6e9ef87f647231a4772af058c277884e6d6e860f27bb25cb9937738a7d94"}
Nov 02 14:55:57.034640 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:57.034036    2577 generic.go:356] "Generic (PLEG): container finished" podID="632223b57cdaa2366b6d306d2b0b8330" containerID="e65cb548c4a51c2f73fae4d1da4ab7dbfe8a734339eac18b2a77e497bfc4f1c2" exitCode=0
Nov 02 14:55:57.034640 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:57.034083    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal" event={"ID":"632223b57cdaa2366b6d306d2b0b8330","Type":"ContainerDied","Data":"e65cb548c4a51c2f73fae4d1da4ab7dbfe8a734339eac18b2a77e497bfc4f1c2"}
Nov 02 14:55:57.277226 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:57.271003    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal" podStartSLOduration=5.270988224 podStartE2EDuration="5.270988224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:57.142416589 +0000 UTC m=+7.637010512" watchObservedRunningTime="2026-11-02 14:55:57.270988224 +0000 UTC m=+7.765582146"
Nov 02 14:55:57.969072 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:57.968460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:57.969072 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:57.968615    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:55:57.969072 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:57.969004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:57.969072 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:57.969027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:55:57.969486 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:57.969116    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:55:57.969486 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:57.969198    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:55:58.044981 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:58.044943    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-spbjm" event={"ID":"c7d93f72-6fc0-469a-8c16-9feed09bda86","Type":"ContainerStarted","Data":"ea03c1630159259458639e0c100ba96398a661b8ccd4013985580b076fb87c85"}
Nov 02 14:55:58.052927 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:58.052875    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal" event={"ID":"632223b57cdaa2366b6d306d2b0b8330","Type":"ContainerStarted","Data":"b50edc609036e72144ffeedc1ae7c935ba74ff9472774fa2a6571dd38b9c0bcc"}
Nov 02 14:55:58.297937 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:58.297792    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-spbjm" podStartSLOduration=3.418707231 podStartE2EDuration="8.297770506s" totalImagesPullingTime="4.879063275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:58.171181536 +0000 UTC m=+8.665775464" watchObservedRunningTime="2026-11-02 14:55:58.297770506 +0000 UTC m=+8.792364436"
Nov 02 14:55:59.066272 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:59.066232    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9htnf" event={"ID":"acede18e-4d8e-47ee-b31f-918658a9ffba","Type":"ContainerStarted","Data":"6d862ec2cfec8fafe557723225be9baa1befa5dbb272dbda7482571c8e6d4913"}
Nov 02 14:55:59.189592 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:59.189534    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9htnf" podStartSLOduration=2.9799069620000003 podStartE2EDuration="9.189514856s" totalImagesPullingTime="6.209607894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:59.1891111 +0000 UTC m=+9.683705027" watchObservedRunningTime="2026-11-02 14:55:59.189514856 +0000 UTC m=+9.684108779"
Nov 02 14:55:59.189997 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:59.189965    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal" podStartSLOduration=8.189958468 podStartE2EDuration="8.189958468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:58.298384411 +0000 UTC m=+8.792978337" watchObservedRunningTime="2026-11-02 14:55:59.189958468 +0000 UTC m=+9.684552392"
Nov 02 14:55:59.631545 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:59.631506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:59.631764 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:59.631743    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:59.631849 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:59.631814    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs podName:9bea213e-5ff8-447d-98db-fa0ec51cda10 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:07.631796694 +0000 UTC m=+18.126390600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs") pod "network-metrics-daemon-rkxt7" (UID: "9bea213e-5ff8-447d-98db-fa0ec51cda10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:59.732847 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:59.732811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:59.733014 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:59.732974    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:59.733087 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:59.733027    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret podName:41cff577-8a79-4b9e-89c8-a93e1ecf4ff0 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:07.73301213 +0000 UTC m=+18.227606044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret") pod "global-pull-secret-syncer-nx6px" (UID: "41cff577-8a79-4b9e-89c8-a93e1ecf4ff0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:59.833441 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:59.833403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw78t\" (UniqueName: \"kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t\") pod \"network-check-target-jjhkj\" (UID: \"8eaf7fe9-c300-4798-8212-e57e5a9cb03d\") " pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:55:59.833633 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:59.833614    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:59.833688 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:59.833638    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:59.833688 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:59.833647    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hw78t for pod openshift-network-diagnostics/network-check-target-jjhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:59.833757 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:59.833692    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t podName:8eaf7fe9-c300-4798-8212-e57e5a9cb03d nodeName:}" failed. No retries permitted until 2026-11-02 14:56:07.833676126 +0000 UTC m=+18.328270042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hw78t" (UniqueName: "kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t") pod "network-check-target-jjhkj" (UID: "8eaf7fe9-c300-4798-8212-e57e5a9cb03d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:59.970150 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:59.969432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:55:59.970150 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:59.969593    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:55:59.970150 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:59.969964    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:55:59.970150 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:55:59.969984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:55:59.970150 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:59.970035    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:55:59.970150 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:55:59.970100    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:56:00.069649 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:00.069561    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t" event={"ID":"49e40d7f-ab6a-4817-ba3e-3e2ab361b454","Type":"ContainerStarted","Data":"f69b8faf4d3881655d81a046b2f8bb07acf94cb2b400eb19813cd36c8d7b9ba2"}
Nov 02 14:56:01.076413 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:01.075811    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w5w2m" event={"ID":"100e3924-bcfc-4208-84ea-c1ff1656e949","Type":"ContainerStarted","Data":"74f8fa9e5d418e230b49d8aa6728437e4eeccf0a32058ed9029726bd5add6d32"}
Nov 02 14:56:01.195970 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:01.195915    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-w5w2m" podStartSLOduration=2.63779035 podStartE2EDuration="11.195896746s" totalImagesPullingTime="8.558106396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:01.194864658 +0000 UTC m=+11.689458581" watchObservedRunningTime="2026-11-02 14:56:01.195896746 +0000 UTC m=+11.690490670"
Nov 02 14:56:01.244754 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:01.244722    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 14:56:01.921209 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:01.921098    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:56:01.244741796Z","UUID":"a62ee6f9-7e87-4178-b89f-ed3d72372214","Handler":null,"Name":"","Endpoint":""}
Nov 02 14:56:01.922797 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:01.922778    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 14:56:01.922797 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:01.922802    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 14:56:01.967207 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:01.967174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:01.967207 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:01.967212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:01.967438 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:01.967186    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:01.967438 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:01.967324    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:56:01.967438 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:01.967392    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:56:01.967597 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:01.967547    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:56:02.081080 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:02.080880    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cllms" event={"ID":"74049a08-892a-4ae2-a5a5-161da6182c1c","Type":"ContainerStarted","Data":"6f407dba060f444bb9d2ff25f011a9a62da17f7a7831965a8fdf9c5d778eed1f"}
Nov 02 14:56:02.082861 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:02.082836    2577 generic.go:356] "Generic (PLEG): container finished" podID="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" containerID="2d81250c08e43e7bd99007969cf73fee39a02a82370c16ed50ec5b2d00010977" exitCode=0
Nov 02 14:56:02.082972 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:02.082922    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-596b4" event={"ID":"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac","Type":"ContainerDied","Data":"2d81250c08e43e7bd99007969cf73fee39a02a82370c16ed50ec5b2d00010977"}
Nov 02 14:56:02.086752 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:02.086281    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t" event={"ID":"49e40d7f-ab6a-4817-ba3e-3e2ab361b454","Type":"ContainerStarted","Data":"0db51850785c5fc70cfb806af222d389fe7b19d9d59d22824f29d98d034f5337"}
Nov 02 14:56:02.197431 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:02.197378    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cllms" podStartSLOduration=2.222926376 podStartE2EDuration="11.197364834s" totalImagesPullingTime="8.974438458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:02.196393114 +0000 UTC m=+12.690987039" watchObservedRunningTime="2026-11-02 14:56:02.197364834 +0000 UTC m=+12.691958755"
Nov 02 14:56:02.570326 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:02.570224    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:56:02.570729 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:02.570711    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:56:03.091745 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:03.091719    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-whl44" event={"ID":"9bb718bc-429a-4875-bd8d-d52dbba0b252","Type":"ContainerStarted","Data":"e8dc6466607410dccae334de11834ac0a04fb3cbdb933b1482dd20228a97f125"}
Nov 02 14:56:03.291345 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:03.291278    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-whl44" podStartSLOduration=4.433893268 podStartE2EDuration="13.291262769s" totalImagesPullingTime="8.857369501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:03.291069127 +0000 UTC m=+13.785663050" watchObservedRunningTime="2026-11-02 14:56:03.291262769 +0000 UTC m=+13.785856693"
Nov 02 14:56:03.967144 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:03.967111    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:03.967339 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:03.967123    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:03.967339 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:03.967238    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:56:03.967339 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:03.967249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:03.967473 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:03.967336    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:56:03.967473 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:03.967423    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:56:04.095854 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:04.095767    2577 generic.go:356] "Generic (PLEG): container finished" podID="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" containerID="b2474e295bfaee035affd5a28a8afda8ad13c822e2d5df4f32fb7b8bd61cf5e9" exitCode=0
Nov 02 14:56:04.096500 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:04.095857    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-596b4" event={"ID":"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac","Type":"ContainerDied","Data":"b2474e295bfaee035affd5a28a8afda8ad13c822e2d5df4f32fb7b8bd61cf5e9"}
Nov 02 14:56:04.098588 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:04.098563    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t" event={"ID":"49e40d7f-ab6a-4817-ba3e-3e2ab361b454","Type":"ContainerStarted","Data":"28435129e4d6c6db423772b26dbc055758a7fe98c62f9f737b2e1918f663366c"}
Nov 02 14:56:04.100362 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:04.100339    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5lhfv" event={"ID":"19ce39d1-e591-4af6-81e4-63d7d7b255c4","Type":"ContainerStarted","Data":"715c64ef5617da3119959dcdd22d621939a0ff7bebb1a399d08678f8c7a48849"}
Nov 02 14:56:04.485511 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:04.485129    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t" podStartSLOduration=3.101997637 podStartE2EDuration="13.485084083s" totalImagesPullingTime="10.383086446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:04.484502601 +0000 UTC m=+14.979096527" watchObservedRunningTime="2026-11-02 14:56:04.485084083 +0000 UTC m=+14.979678006"
Nov 02 14:56:04.604992 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:04.604941    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5lhfv" podStartSLOduration=2.157738719 podStartE2EDuration="13.604928104s" totalImagesPullingTime="11.447189385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:04.604619647 +0000 UTC m=+15.099213568" watchObservedRunningTime="2026-11-02 14:56:04.604928104 +0000 UTC m=+15.099522027"
Nov 02 14:56:05.966886 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:05.966853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:05.966886 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:05.966864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:05.967614 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:05.966982    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:56:05.967614 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:05.967074    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:56:05.967614 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:05.967110    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:05.967614 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:05.967207    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:56:06.107681 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:06.107479    2577 generic.go:356] "Generic (PLEG): container finished" podID="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" containerID="f13140e6b9cc2cf72e46f17da6f3d3cdd6390c4b36a8f49e2e9af8c0c358557b" exitCode=0
Nov 02 14:56:06.107681 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:06.107573    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-596b4" event={"ID":"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac","Type":"ContainerDied","Data":"f13140e6b9cc2cf72e46f17da6f3d3cdd6390c4b36a8f49e2e9af8c0c358557b"}
Nov 02 14:56:07.695883 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:07.695852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:07.696191 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:07.695961    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:56:07.696191 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:07.696013    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs podName:9bea213e-5ff8-447d-98db-fa0ec51cda10 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:23.696000851 +0000 UTC m=+34.190594752 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs") pod "network-metrics-daemon-rkxt7" (UID: "9bea213e-5ff8-447d-98db-fa0ec51cda10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:56:07.797111 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:07.797047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:07.797227 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:07.797177    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:56:07.797285 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:07.797241    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret podName:41cff577-8a79-4b9e-89c8-a93e1ecf4ff0 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:23.797221779 +0000 UTC m=+34.291815682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret") pod "global-pull-secret-syncer-nx6px" (UID: "41cff577-8a79-4b9e-89c8-a93e1ecf4ff0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:56:07.813102 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:07.813071    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:56:07.813275 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:07.813260    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:56:07.814071 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:07.813958    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:56:07.898032 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:07.897957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw78t\" (UniqueName: \"kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t\") pod \"network-check-target-jjhkj\" (UID: \"8eaf7fe9-c300-4798-8212-e57e5a9cb03d\") " pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:07.898135 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:07.898121    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:56:07.898199 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:07.898144    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:56:07.898199 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:07.898155    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hw78t for pod openshift-network-diagnostics/network-check-target-jjhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:56:07.898298 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:07.898203    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t podName:8eaf7fe9-c300-4798-8212-e57e5a9cb03d nodeName:}" failed. No retries permitted until 2026-11-02 14:56:23.898186863 +0000 UTC m=+34.392780771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hw78t" (UniqueName: "kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t") pod "network-check-target-jjhkj" (UID: "8eaf7fe9-c300-4798-8212-e57e5a9cb03d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:56:07.967186 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:07.967161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:07.967186 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:07.967187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:07.967336 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:07.967262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:07.967397 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:07.967370    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:56:07.967500 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:07.967471    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:56:07.967600 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:07.967578    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:56:08.116143 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:08.116112    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8822k" event={"ID":"6f2ebec5-9906-438f-b2ce-010345fa49ba","Type":"ContainerStarted","Data":"e0ceb9b13ad195da98da3767e9d00d2bab2a59371c6ad4c2c1a831369747217f"}
Nov 02 14:56:08.116300 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:08.116178    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8822k" event={"ID":"6f2ebec5-9906-438f-b2ce-010345fa49ba","Type":"ContainerStarted","Data":"c171cd0dbb2305b6c8381d127c6f91b412899097155b90b3490f7047e54d7796"}
Nov 02 14:56:08.116300 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:08.116212    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8822k" event={"ID":"6f2ebec5-9906-438f-b2ce-010345fa49ba","Type":"ContainerStarted","Data":"ce183ac01addddf5a4529d97f49af689b552ab08663f621cdb0b4822077f1464"}
Nov 02 14:56:08.116300 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:08.116249    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8822k" event={"ID":"6f2ebec5-9906-438f-b2ce-010345fa49ba","Type":"ContainerStarted","Data":"af82e78a732a617bad94c2794a556d8149689e4a6483bd26878946eba455b1c6"}
Nov 02 14:56:08.116300 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:08.116297    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8822k" event={"ID":"6f2ebec5-9906-438f-b2ce-010345fa49ba","Type":"ContainerStarted","Data":"78c89371cdb0d6e9ec3a4208b56bcdff77ccbc1e91f51d3f3ad2388a6223f62c"}
Nov 02 14:56:08.116513 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:08.116359    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8822k" event={"ID":"6f2ebec5-9906-438f-b2ce-010345fa49ba","Type":"ContainerStarted","Data":"8b69e79714480c1f66f1d42d79add4b5f6957f974ce2a025aadc81aa5a8c3e82"}
Nov 02 14:56:09.120462 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:09.120432    2577 generic.go:356] "Generic (PLEG): container finished" podID="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" containerID="a988f4d8c1a9eb03c33c6d85dc02da88d5616e0cc5a0285e669bfa641a591559" exitCode=0
Nov 02 14:56:09.120847 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:09.120476    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-596b4" event={"ID":"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac","Type":"ContainerDied","Data":"a988f4d8c1a9eb03c33c6d85dc02da88d5616e0cc5a0285e669bfa641a591559"}
Nov 02 14:56:09.968271 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:09.968237    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:09.968424 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:09.968365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:09.968424 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:09.968372    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:56:09.968495 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:09.968458    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:56:09.968531 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:09.968507    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:09.968582 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:09.968558    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:56:10.125793 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:10.125763    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8822k" event={"ID":"6f2ebec5-9906-438f-b2ce-010345fa49ba","Type":"ContainerStarted","Data":"a2f0ec01cdd764c96ad86ede83b9645876ee81d1f81fc643ac28066e50acb7f4"}
Nov 02 14:56:10.128924 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:10.128900    2577 generic.go:356] "Generic (PLEG): container finished" podID="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" containerID="ef6d4ace3327cea9913c8bdf3e7b860c4d9bfb43128480d3666ccc19c3d04016" exitCode=0
Nov 02 14:56:10.129149 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:10.128940    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-596b4" event={"ID":"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac","Type":"ContainerDied","Data":"ef6d4ace3327cea9913c8bdf3e7b860c4d9bfb43128480d3666ccc19c3d04016"}
Nov 02 14:56:11.134448 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:11.134414    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-596b4" event={"ID":"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac","Type":"ContainerStarted","Data":"ca8a9f67b31e39d17769444bdbf1bc6be4947585e5d9f29d40b3d41181689832"}
Nov 02 14:56:11.354998 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:11.354943    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-596b4" podStartSLOduration=8.138673981 podStartE2EDuration="21.354927972s" totalImagesPullingTime="13.216253991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:11.304851632 +0000 UTC m=+21.799445554" watchObservedRunningTime="2026-11-02 14:56:11.354927972 +0000 UTC m=+21.849521894"
Nov 02 14:56:11.967670 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:11.967635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:11.967866 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:11.967635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:11.967866 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:11.967743    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:56:11.967866 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:11.967840    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:56:11.968003 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:11.967897    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:11.968003 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:11.967960    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:56:13.142452 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:13.142411    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8822k" event={"ID":"6f2ebec5-9906-438f-b2ce-010345fa49ba","Type":"ContainerStarted","Data":"93ee56f8a8b0140afb9bb0f4f9ba337002804facd287249957279a1c01c89e4e"}
Nov 02 14:56:13.143008 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:13.142641    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:13.158377 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:13.158351    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:13.418202 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:13.418075    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8822k" podStartSLOduration=8.059853294 podStartE2EDuration="23.418060987s" totalImagesPullingTime="15.358207693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:13.393734694 +0000 UTC m=+23.888328617" watchObservedRunningTime="2026-11-02 14:56:13.418060987 +0000 UTC m=+23.912654909"
Nov 02 14:56:13.966720 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:13.966692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:13.966720 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:13.966719    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:13.966995 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:13.966753    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:13.966995 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:13.966832    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:56:13.966995 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:13.966882    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:56:13.966995 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:13.966973    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:56:14.144282 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:14.144257    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:56:14.144703 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:14.144685    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:14.159176 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:14.159155    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:14.786570 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:14.786537    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nx6px"]
Nov 02 14:56:14.786734 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:14.786673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:14.786779 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:14.786761    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:56:14.789767 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:14.789736    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jjhkj"]
Nov 02 14:56:14.789903 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:14.789852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:14.789981 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:14.789951    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:56:14.790390 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:14.790359    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rkxt7"]
Nov 02 14:56:14.790493 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:14.790478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:14.790603 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:14.790584    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:56:15.146025 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:15.145950    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:56:15.425537 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:15.425449    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:15.967292 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:15.967266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:15.967434 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:15.967397    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:56:16.967625 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:16.967593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:16.968098 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:16.967711    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:56:16.968098 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:16.967776    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:16.968098 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:16.967885    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:56:17.161217 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:17.161128    2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8822k" podUID="6f2ebec5-9906-438f-b2ce-010345fa49ba" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 02 14:56:17.967474 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:17.967442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:17.967753 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:17.967571    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:56:18.966848 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:18.966813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:18.967005 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:18.966961    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhkj" podUID="8eaf7fe9-c300-4798-8212-e57e5a9cb03d"
Nov 02 14:56:18.967053 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:18.967005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:18.967105 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:18.967089    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkxt7" podUID="9bea213e-5ff8-447d-98db-fa0ec51cda10"
Nov 02 14:56:19.968600 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:19.968566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:19.968994 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:19.968691    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nx6px" podUID="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0"
Nov 02 14:56:20.804429 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.804192    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeReady"
Nov 02 14:56:20.804562 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.804478    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 14:56:20.922811 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.922774    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v4f7p"]
Nov 02 14:56:20.924496 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.924475    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-684675bd5f-zwwbp"]
Nov 02 14:56:20.924628 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.924609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-v4f7p"
Nov 02 14:56:20.925820 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.925805    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-64bm4"]
Nov 02 14:56:20.925926 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.925914    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:20.927495 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.927478    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xx5tn"]
Nov 02 14:56:20.927595 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.927580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-64bm4"
Nov 02 14:56:20.929101 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.929088    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xx5tn"
Nov 02 14:56:20.947442 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.947422    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"]
Nov 02 14:56:20.948928 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.948909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:20.967326 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.967293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:20.967485 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.967467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:20.972140 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.972060    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74fd4fbb69-thwc5"]
Nov 02 14:56:20.974081 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.974060    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"]
Nov 02 14:56:20.974208 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.974189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:20.975681 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.975666    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"]
Nov 02 14:56:20.975790 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.975769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:20.977420 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.977403    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"]
Nov 02 14:56:20.977524 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.977510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:20.978782 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.978768    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-ggtvw"]
Nov 02 14:56:20.978885 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.978872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:20.980447 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.980432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:20.992157 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.992139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psj4b\" (UniqueName: \"kubernetes.io/projected/1bcba632-1434-4533-951f-84b8bba1f0d8-kube-api-access-psj4b\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:20.992242 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.992176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdhh\" (UniqueName: \"kubernetes.io/projected/f25b0184-4053-4694-acc1-c26e9f5602aa-kube-api-access-khdhh\") pod \"kas-connection-checker-685895c7c9-v4f7p\" (UID: \"f25b0184-4053-4694-acc1-c26e9f5602aa\") " pod="kube-system/kas-connection-checker-685895c7c9-v4f7p"
Nov 02 14:56:20.992242 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.992197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flmc9\" (UniqueName: \"kubernetes.io/projected/0290bc25-3e34-4d43-9215-f15cfc44f686-kube-api-access-flmc9\") pod \"kas-connection-checker-685895c7c9-xx5tn\" (UID: \"0290bc25-3e34-4d43-9215-f15cfc44f686\") " pod="kube-system/kas-connection-checker-685895c7c9-xx5tn"
Nov 02 14:56:20.992242 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.992212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:20.992436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.992299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:20.992436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.992382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-stats-auth\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:20.992436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.992407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg79x\" (UniqueName: \"kubernetes.io/projected/63faa28d-8806-4332-99c4-9f715a0b1a86-kube-api-access-kg79x\") pod \"kas-connection-checker-685895c7c9-64bm4\" (UID: \"63faa28d-8806-4332-99c4-9f715a0b1a86\") " pod="kube-system/kas-connection-checker-685895c7c9-64bm4"
Nov 02 14:56:20.992436 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:20.992431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-default-certificate\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:21.019390 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.019368    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dcff7b7b8-bck5q"]
Nov 02 14:56:21.020939 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.020921    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.021716 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.021698    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"]
Nov 02 14:56:21.023306 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.023290    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q"]
Nov 02 14:56:21.023424 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.023405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.025186 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.025167    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l"]
Nov 02 14:56:21.025292 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.025282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q"
Nov 02 14:56:21.026888 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.026872    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"]
Nov 02 14:56:21.027079 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.027057    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l"
Nov 02 14:56:21.028503 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.028482    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f6fgv"]
Nov 02 14:56:21.028770 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.028756    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:21.030556 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.030539    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jcqwk"]
Nov 02 14:56:21.030653 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.030645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:21.035062 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.034977    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v4f7p"]
Nov 02 14:56:21.035158 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.035129    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jcqwk"
Nov 02 14:56:21.069982 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.069955    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 14:56:21.069982 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.069977    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 14:56:21.070176 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.070013    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 14:56:21.070176 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.070031    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 14:56:21.070176 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.070060    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 14:56:21.070176 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.070096    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 14:56:21.071092 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.071071    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-64bm4"]
Nov 02 14:56:21.071793 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.071775    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-684675bd5f-zwwbp"]
Nov 02 14:56:21.072606 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.072561    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"]
Nov 02 14:56:21.073497 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.073479    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"]
Nov 02 14:56:21.093027 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.092990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:21.093171 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc12a767-3259-4dc0-a809-d123b58334e3-config\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:21.093171 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-config-volume\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:21.093326 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg79x\" (UniqueName: \"kubernetes.io/projected/63faa28d-8806-4332-99c4-9f715a0b1a86-kube-api-access-kg79x\") pod \"kas-connection-checker-685895c7c9-64bm4\" (UID: \"63faa28d-8806-4332-99c4-9f715a0b1a86\") " pod="kube-system/kas-connection-checker-685895c7c9-64bm4"
Nov 02 14:56:21.093465 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-certificates\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.093545 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftvbk\" (UniqueName: \"kubernetes.io/projected/063ba34d-7544-410f-b861-5c300cc8f6d4-kube-api-access-ftvbk\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.093545 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9tf9\" (UniqueName: \"kubernetes.io/projected/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-kube-api-access-z9tf9\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:21.093545 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkkc\" (UniqueName: \"kubernetes.io/projected/9fb94c55-ff22-4e24-99da-047c765a8598-kube-api-access-xvkkc\") pod \"ingress-canary-jcqwk\" (UID: \"9fb94c55-ff22-4e24-99da-047c765a8598\") " pod="openshift-ingress-canary/ingress-canary-jcqwk"
Nov 02 14:56:21.093662 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/234a4a99-5491-44ef-b58c-3e4ab32584e5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:21.093703 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-default-certificate\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:21.093737 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtqv7\" (UniqueName: \"kubernetes.io/projected/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-kube-api-access-dtqv7\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:21.093737 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/063ba34d-7544-410f-b861-5c300cc8f6d4-service-ca-bundle\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.093821 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf04dbe2-4354-4849-959c-33002c7ece68-trusted-ca\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.093867 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb39336d-67c0-4cf2-aaff-c873d361f9b2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:21.093906 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/063ba34d-7544-410f-b861-5c300cc8f6d4-snapshots\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.093948 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf04dbe2-4354-4849-959c-33002c7ece68-serving-cert\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.093997 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.093969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/234a4a99-5491-44ef-b58c-3e4ab32584e5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:21.094061 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flmc9\" (UniqueName: \"kubernetes.io/projected/0290bc25-3e34-4d43-9215-f15cfc44f686-kube-api-access-flmc9\") pod \"kas-connection-checker-685895c7c9-xx5tn\" (UID: \"0290bc25-3e34-4d43-9215-f15cfc44f686\") " pod="kube-system/kas-connection-checker-685895c7c9-xx5tn"
Nov 02 14:56:21.094113 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:21.094113 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-tmp-dir\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:21.094209 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mls\" (UniqueName: \"kubernetes.io/projected/be0e9509-693a-4b99-b5c5-32770e758b5a-kube-api-access-64mls\") pod \"network-check-source-6f78d59f99-cnq6q\" (UID: \"be0e9509-693a-4b99-b5c5-32770e758b5a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q"
Nov 02 14:56:21.094209 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/885145c5-6077-4435-89ce-73af1e6a5600-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:21.094209 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063ba34d-7544-410f-b861-5c300cc8f6d4-tmp\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.094374 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb94c55-ff22-4e24-99da-047c765a8598-cert\") pod \"ingress-canary-jcqwk\" (UID: \"9fb94c55-ff22-4e24-99da-047c765a8598\") " pod="openshift-ingress-canary/ingress-canary-jcqwk"
Nov 02 14:56:21.094374 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.094374 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-bound-sa-token\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.094530 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:21.094530 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-stats-auth\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:21.094607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d2ae30e-0191-41ff-85da-dedf17ad27b2-ca-trust-extracted\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.094655 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.094639    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:21.594621467 +0000 UTC m=+32.089215384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : configmap references non-existent config key: service-ca.crt
Nov 02 14:56:21.094830 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc12a767-3259-4dc0-a809-d123b58334e3-serving-cert\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:21.094905 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:21.094905 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-image-registry-private-configuration\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.094990 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshc2\" (UniqueName: \"kubernetes.io/projected/885145c5-6077-4435-89ce-73af1e6a5600-kube-api-access-cshc2\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:21.094990 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:21.095075 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.094995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-image-registry-private-configuration\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.095075 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf04dbe2-4354-4849-959c-33002c7ece68-config\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.095159 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2znqw\" (UniqueName: \"kubernetes.io/projected/234a4a99-5491-44ef-b58c-3e4ab32584e5-kube-api-access-2znqw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:21.095159 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/063ba34d-7544-410f-b861-5c300cc8f6d4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.095159 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-trusted-ca\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.095159 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klllg\" (UniqueName: \"kubernetes.io/projected/fc12a767-3259-4dc0-a809-d123b58334e3-kube-api-access-klllg\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:21.095350 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-bound-sa-token\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.095350 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-ca-trust-extracted\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.095350 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:21.095350 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psj4b\" (UniqueName: \"kubernetes.io/projected/1bcba632-1434-4533-951f-84b8bba1f0d8-kube-api-access-psj4b\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:21.095350 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdhh\" (UniqueName: \"kubernetes.io/projected/f25b0184-4053-4694-acc1-c26e9f5602aa-kube-api-access-khdhh\") pod \"kas-connection-checker-685895c7c9-v4f7p\" (UID: \"f25b0184-4053-4694-acc1-c26e9f5602aa\") " pod="kube-system/kas-connection-checker-685895c7c9-v4f7p"
Nov 02 14:56:21.095350 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:21.095350 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwhf\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-kube-api-access-2fwhf\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.095350 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-trusted-ca\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.095350 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.095306    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:56:21.095350 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.095778 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-installation-pull-secrets\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.095778 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.095364    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:21.595353488 +0000 UTC m=+32.089947388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : secret "router-metrics-certs-default" not found
Nov 02 14:56:21.095778 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l647\" (UniqueName: \"kubernetes.io/projected/bf04dbe2-4354-4849-959c-33002c7ece68-kube-api-access-9l647\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.095778 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87hwb\" (UniqueName: \"kubernetes.io/projected/27c6485e-f8bc-4b94-839c-746297629798-kube-api-access-87hwb\") pod \"volume-data-source-validator-58c9f5f46d-96v9l\" (UID: \"27c6485e-f8bc-4b94-839c-746297629798\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l"
Nov 02 14:56:21.095778 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/063ba34d-7544-410f-b861-5c300cc8f6d4-serving-cert\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.095778 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbskj\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-kube-api-access-qbskj\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.095778 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-installation-pull-secrets\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.095778 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.095698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-certificates\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.097923 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.097906    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-default-certificate\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:21.098026 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.097956    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-stats-auth\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:21.120464 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.120437    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-ggtvw"]
Nov 02 14:56:21.121041 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.121025    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"]
Nov 02 14:56:21.121578 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.121565    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q"]
Nov 02 14:56:21.122091 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.122079    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jcqwk"]
Nov 02 14:56:21.122677 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.122662    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f6fgv"]
Nov 02 14:56:21.123183 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.123173    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xx5tn"]
Nov 02 14:56:21.123779 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.123760    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"]
Nov 02 14:56:21.144989 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.144964    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:21.144989 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.144964    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 14:56:21.167616 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.167589    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:21.169116 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.169100    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 14:56:21.169375 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.169338    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 14:56:21.196912 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.196881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.197003 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.196926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-bound-sa-token\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.197003 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.196983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d2ae30e-0191-41ff-85da-dedf17ad27b2-ca-trust-extracted\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.197089 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc12a767-3259-4dc0-a809-d123b58334e3-serving-cert\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:21.197089 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:21.197089 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-image-registry-private-configuration\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.197187 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshc2\" (UniqueName: \"kubernetes.io/projected/885145c5-6077-4435-89ce-73af1e6a5600-kube-api-access-cshc2\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:21.197187 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:21.197187 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-image-registry-private-configuration\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.197281 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf04dbe2-4354-4849-959c-33002c7ece68-config\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.197281 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2znqw\" (UniqueName: \"kubernetes.io/projected/234a4a99-5491-44ef-b58c-3e4ab32584e5-kube-api-access-2znqw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:21.197363 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/063ba34d-7544-410f-b861-5c300cc8f6d4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.197404 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-trusted-ca\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.197404 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klllg\" (UniqueName: \"kubernetes.io/projected/fc12a767-3259-4dc0-a809-d123b58334e3-kube-api-access-klllg\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:21.197497 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-bound-sa-token\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.197497 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197433    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d2ae30e-0191-41ff-85da-dedf17ad27b2-ca-trust-extracted\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.197497 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-ca-trust-extracted\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.197497 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:21.197687 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwhf\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-kube-api-access-2fwhf\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.197687 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-trusted-ca\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.197687 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.197687 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-installation-pull-secrets\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.197687 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l647\" (UniqueName: \"kubernetes.io/projected/bf04dbe2-4354-4849-959c-33002c7ece68-kube-api-access-9l647\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.197875 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87hwb\" (UniqueName: \"kubernetes.io/projected/27c6485e-f8bc-4b94-839c-746297629798-kube-api-access-87hwb\") pod \"volume-data-source-validator-58c9f5f46d-96v9l\" (UID: \"27c6485e-f8bc-4b94-839c-746297629798\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l"
Nov 02 14:56:21.197875 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/063ba34d-7544-410f-b861-5c300cc8f6d4-serving-cert\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.197875 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbskj\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-kube-api-access-qbskj\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.197875 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-installation-pull-secrets\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.197875 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197772    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-ca-trust-extracted\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.197875 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-certificates\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.197875 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:21.197875 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc12a767-3259-4dc0-a809-d123b58334e3-config\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-config-volume\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-certificates\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftvbk\" (UniqueName: \"kubernetes.io/projected/063ba34d-7544-410f-b861-5c300cc8f6d4-kube-api-access-ftvbk\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9tf9\" (UniqueName: \"kubernetes.io/projected/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-kube-api-access-z9tf9\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.197973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkkc\" (UniqueName: \"kubernetes.io/projected/9fb94c55-ff22-4e24-99da-047c765a8598-kube-api-access-xvkkc\") pod \"ingress-canary-jcqwk\" (UID: \"9fb94c55-ff22-4e24-99da-047c765a8598\") " pod="openshift-ingress-canary/ingress-canary-jcqwk"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/234a4a99-5491-44ef-b58c-3e4ab32584e5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtqv7\" (UniqueName: \"kubernetes.io/projected/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-kube-api-access-dtqv7\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/063ba34d-7544-410f-b861-5c300cc8f6d4-service-ca-bundle\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf04dbe2-4354-4849-959c-33002c7ece68-trusted-ca\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb39336d-67c0-4cf2-aaff-c873d361f9b2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/063ba34d-7544-410f-b861-5c300cc8f6d4-snapshots\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf04dbe2-4354-4849-959c-33002c7ece68-serving-cert\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/234a4a99-5491-44ef-b58c-3e4ab32584e5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:21.198302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-tmp-dir\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:21.198924 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mls\" (UniqueName: \"kubernetes.io/projected/be0e9509-693a-4b99-b5c5-32770e758b5a-kube-api-access-64mls\") pod \"network-check-source-6f78d59f99-cnq6q\" (UID: \"be0e9509-693a-4b99-b5c5-32770e758b5a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q"
Nov 02 14:56:21.198924 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/885145c5-6077-4435-89ce-73af1e6a5600-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:21.198924 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063ba34d-7544-410f-b861-5c300cc8f6d4-tmp\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.198924 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198403    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-certificates\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.198924 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb94c55-ff22-4e24-99da-047c765a8598-cert\") pod \"ingress-canary-jcqwk\" (UID: \"9fb94c55-ff22-4e24-99da-047c765a8598\") " pod="openshift-ingress-canary/ingress-canary-jcqwk"
Nov 02 14:56:21.198924 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198454    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-certificates\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.198924 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198647    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-tmp-dir\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:21.198924 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198717    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063ba34d-7544-410f-b861-5c300cc8f6d4-tmp\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.198924 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.198757    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/063ba34d-7544-410f-b861-5c300cc8f6d4-snapshots\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.218575 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.218529    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:21.218797 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.218784    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 14:56:21.229294 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.229273    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:21.244307 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.244289    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 14:56:21.266725 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.266701    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 14:56:21.269074 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.269060    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 14:56:21.269074 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.269069    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 14:56:21.278930 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.278908    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc12a767-3259-4dc0-a809-d123b58334e3-config\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:21.317722 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.317632    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:21.317722 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.317650    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 14:56:21.317899 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.317886    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 14:56:21.318511 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.318498    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 14:56:21.318551 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.318505    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:21.318887 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.318869    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/063ba34d-7544-410f-b861-5c300cc8f6d4-service-ca-bundle\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.318966 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.318950    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb39336d-67c0-4cf2-aaff-c873d361f9b2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:21.343427 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.343401    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 14:56:21.367522 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.367494    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:21.416590 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.416562    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 14:56:21.416947 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.416930    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 14:56:21.417975 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.417957    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf04dbe2-4354-4849-959c-33002c7ece68-config\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.419042 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.419024    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/885145c5-6077-4435-89ce-73af1e6a5600-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:21.603402 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.603304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:21.603543 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.603465    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:56:21.603543 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.603492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:21.603645 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.603543    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:22.603523624 +0000 UTC m=+33.098117541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : secret "router-metrics-certs-default" not found
Nov 02 14:56:21.603716 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.603685    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:22.603661821 +0000 UTC m=+33.098255736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : configmap references non-existent config key: service-ca.crt
Nov 02 14:56:21.714812 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.714784    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 14:56:21.715117 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.715094    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 14:56:21.715607 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.715587    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 14:56:21.717932 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.717767    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:21.718047 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.718033    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcff7b7b8-bck5q: secret "image-registry-tls" not found
Nov 02 14:56:21.718095 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.718074    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:21.718144 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.718105    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls podName:fac285e0-f60a-4f53-a7e8-dee42bb83a0a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:22.218077973 +0000 UTC m=+32.712671893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls") pod "image-registry-dcff7b7b8-bck5q" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a") : secret "image-registry-tls" not found
Nov 02 14:56:21.721503 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.718225    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74fd4fbb69-thwc5: secret "image-registry-tls" not found
Nov 02 14:56:21.721503 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.718297    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls podName:8d2ae30e-0191-41ff-85da-dedf17ad27b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:22.218266111 +0000 UTC m=+32.712860029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls") pod "image-registry-74fd4fbb69-thwc5" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2") : secret "image-registry-tls" not found
Nov 02 14:56:21.721503 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.718920    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-trusted-ca\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.721503 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.719011    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-trusted-ca\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.721503 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.719784    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf04dbe2-4354-4849-959c-33002c7ece68-trusted-ca\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.764167 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.764093    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 14:56:21.764167 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.764090    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 14:56:21.764167 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.764145    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 14:56:21.764167 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.764145    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 14:56:21.764503 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.764198    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 14:56:21.764503 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.764394    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:21.764903 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.764826    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 14:56:21.764903 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.764863    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 14:56:21.765108 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.764911    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:21.765177 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.765133    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 14:56:21.765213 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.765174    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:21.765213 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.765182    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:21.765213 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.765182    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 14:56:21.765734 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.765716    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"]
Nov 02 14:56:21.767361 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.767345    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:21.767426 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.767408    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls podName:885145c5-6077-4435-89ce-73af1e6a5600 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:22.267386888 +0000 UTC m=+32.761980791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-q2t8x" (UID: "885145c5-6077-4435-89ce-73af1e6a5600") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:21.767871 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.767854    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:56:21.767952 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.767912    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert podName:fb39336d-67c0-4cf2-aaff-c873d361f9b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:22.267895417 +0000 UTC m=+32.762489334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mf5tv" (UID: "fb39336d-67c0-4cf2-aaff-c873d361f9b2") : secret "networking-console-plugin-cert" not found
Nov 02 14:56:21.768444 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.768425    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:56:21.768566 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.768488    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls podName:7ff63dfc-17d4-4efc-a23a-d2e46a6885fc nodeName:}" failed. No retries permitted until 2026-11-02 14:56:22.268470869 +0000 UTC m=+32.763064789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vhx5m" (UID: "7ff63dfc-17d4-4efc-a23a-d2e46a6885fc") : secret "samples-operator-tls" not found
Nov 02 14:56:21.768897 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.768871    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/234a4a99-5491-44ef-b58c-3e4ab32584e5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:21.769666 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.769638    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc12a767-3259-4dc0-a809-d123b58334e3-serving-cert\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:21.769811 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.769791    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-image-registry-private-configuration\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.769872 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.769839    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-image-registry-private-configuration\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.770237 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.770219    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/063ba34d-7544-410f-b861-5c300cc8f6d4-serving-cert\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.770755 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.770737    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/234a4a99-5491-44ef-b58c-3e4ab32584e5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:21.790671 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.790645    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 14:56:21.800751 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.800720    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf04dbe2-4354-4849-959c-33002c7ece68-serving-cert\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.814571 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.814547    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 14:56:21.814676 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.814586    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 14:56:21.814676 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.814595    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 14:56:21.814676 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.814586    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 14:56:21.814904 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.814892    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 14:56:21.818057 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.818043    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:56:21.818118 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:21.818093    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls podName:b639a9d2-f451-4cb1-a31f-c3de10b8d3c1 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:22.318078694 +0000 UTC m=+32.812672593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls") pod "dns-default-f6fgv" (UID: "b639a9d2-f451-4cb1-a31f-c3de10b8d3c1") : secret "dns-default-metrics-tls" not found
Nov 02 14:56:21.818405 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.818388    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-config-volume\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:21.821075 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.821052    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb94c55-ff22-4e24-99da-047c765a8598-cert\") pod \"ingress-canary-jcqwk\" (UID: \"9fb94c55-ff22-4e24-99da-047c765a8598\") " pod="openshift-ingress-canary/ingress-canary-jcqwk"
Nov 02 14:56:21.842635 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.842608    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg79x\" (UniqueName: \"kubernetes.io/projected/63faa28d-8806-4332-99c4-9f715a0b1a86-kube-api-access-kg79x\") pod \"kas-connection-checker-685895c7c9-64bm4\" (UID: \"63faa28d-8806-4332-99c4-9f715a0b1a86\") " pod="kube-system/kas-connection-checker-685895c7c9-64bm4"
Nov 02 14:56:21.842723 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.842638    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdhh\" (UniqueName: \"kubernetes.io/projected/f25b0184-4053-4694-acc1-c26e9f5602aa-kube-api-access-khdhh\") pod \"kas-connection-checker-685895c7c9-v4f7p\" (UID: \"f25b0184-4053-4694-acc1-c26e9f5602aa\") " pod="kube-system/kas-connection-checker-685895c7c9-v4f7p"
Nov 02 14:56:21.842723 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.842656    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flmc9\" (UniqueName: \"kubernetes.io/projected/0290bc25-3e34-4d43-9215-f15cfc44f686-kube-api-access-flmc9\") pod \"kas-connection-checker-685895c7c9-xx5tn\" (UID: \"0290bc25-3e34-4d43-9215-f15cfc44f686\") " pod="kube-system/kas-connection-checker-685895c7c9-xx5tn"
Nov 02 14:56:21.842723 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.842663    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psj4b\" (UniqueName: \"kubernetes.io/projected/1bcba632-1434-4533-951f-84b8bba1f0d8-kube-api-access-psj4b\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:21.849752 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.849738    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-64bm4"
Nov 02 14:56:21.856268 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.856226    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xx5tn"
Nov 02 14:56:21.867121 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.867092    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbskj\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-kube-api-access-qbskj\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.867228 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.867121    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-bound-sa-token\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.867618 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.867295    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l647\" (UniqueName: \"kubernetes.io/projected/bf04dbe2-4354-4849-959c-33002c7ece68-kube-api-access-9l647\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.867727 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.867709    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwhf\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-kube-api-access-2fwhf\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.867790 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.867756    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2znqw\" (UniqueName: \"kubernetes.io/projected/234a4a99-5491-44ef-b58c-3e4ab32584e5-kube-api-access-2znqw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:21.867845 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.867819    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5rstk\""
Nov 02 14:56:21.867894 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.867853    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-p249k\""
Nov 02 14:56:21.867945 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.867893    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-b2j82\""
Nov 02 14:56:21.867945 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.867925    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nw765\""
Nov 02 14:56:21.868915 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.868331    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klllg\" (UniqueName: \"kubernetes.io/projected/fc12a767-3259-4dc0-a809-d123b58334e3-kube-api-access-klllg\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:21.868915 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.868354    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4ks69\""
Nov 02 14:56:21.868915 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.868503    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ptvxg\""
Nov 02 14:56:21.868915 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.868561    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87hwb\" (UniqueName: \"kubernetes.io/projected/27c6485e-f8bc-4b94-839c-746297629798-kube-api-access-87hwb\") pod \"volume-data-source-validator-58c9f5f46d-96v9l\" (UID: \"27c6485e-f8bc-4b94-839c-746297629798\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l"
Nov 02 14:56:21.868915 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.868577    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-jb9s2\""
Nov 02 14:56:21.869202 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.869149    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshc2\" (UniqueName: \"kubernetes.io/projected/885145c5-6077-4435-89ce-73af1e6a5600-kube-api-access-cshc2\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:21.870209 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.870182    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-bound-sa-token\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.870729 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.870681    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftvbk\" (UniqueName: \"kubernetes.io/projected/063ba34d-7544-410f-b861-5c300cc8f6d4-kube-api-access-ftvbk\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:21.871292 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.871262    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9tf9\" (UniqueName: \"kubernetes.io/projected/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-kube-api-access-z9tf9\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:21.871720 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.871699    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtqv7\" (UniqueName: \"kubernetes.io/projected/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-kube-api-access-dtqv7\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:21.871804 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.871789    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mls\" (UniqueName: \"kubernetes.io/projected/be0e9509-693a-4b99-b5c5-32770e758b5a-kube-api-access-64mls\") pod \"network-check-source-6f78d59f99-cnq6q\" (UID: \"be0e9509-693a-4b99-b5c5-32770e758b5a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q"
Nov 02 14:56:21.871963 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.871942    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkkc\" (UniqueName: \"kubernetes.io/projected/9fb94c55-ff22-4e24-99da-047c765a8598-kube-api-access-xvkkc\") pod \"ingress-canary-jcqwk\" (UID: \"9fb94c55-ff22-4e24-99da-047c765a8598\") " pod="openshift-ingress-canary/ingress-canary-jcqwk"
Nov 02 14:56:21.891573 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.891497    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74fd4fbb69-thwc5"]
Nov 02 14:56:21.914329 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.914027    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mnrgm\""
Nov 02 14:56:21.914329 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.914103    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9nw68\""
Nov 02 14:56:21.914329 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.914192    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-44bxs\""
Nov 02 14:56:21.914329 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.914268    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rtl8l\""
Nov 02 14:56:21.917334 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.917210    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bm8rt\""
Nov 02 14:56:21.917334 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.917257    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wtdgk\""
Nov 02 14:56:21.917591 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.917576    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 14:56:21.920706 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.920682    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-installation-pull-secrets\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:21.921446 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.921424    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-installation-pull-secrets\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:21.943460 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.943437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:21.950284 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.950162    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q"
Nov 02 14:56:21.955898 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.955872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l"
Nov 02 14:56:21.961691 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.961667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:21.966542 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.965173    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dcff7b7b8-bck5q"]
Nov 02 14:56:21.968547 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.967107    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:21.975675 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:21.975651    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jcqwk"
Nov 02 14:56:22.017209 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.016604    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 14:56:22.017209 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.016791    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l"]
Nov 02 14:56:22.018347 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.018198    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"]
Nov 02 14:56:22.019071 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.019053    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/063ba34d-7544-410f-b861-5c300cc8f6d4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:22.042448 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.041627    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 14:56:22.042448 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.041701    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dh847\""
Nov 02 14:56:22.138140 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.138075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-v4f7p"
Nov 02 14:56:22.160514 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.160495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:22.216974 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.216772    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:22.234856 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.234831    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xx5tn"]
Nov 02 14:56:22.238835 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:56:22.238806    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0290bc25_3e34_4d43_9215_f15cfc44f686.slice/crio-35549c1fd97c78be1f742ab6a916a99068c9e14728cf801854e852ccda8fa0f8 WatchSource:0}: Error finding container 35549c1fd97c78be1f742ab6a916a99068c9e14728cf801854e852ccda8fa0f8: Status 404 returned error can't find the container with id 35549c1fd97c78be1f742ab6a916a99068c9e14728cf801854e852ccda8fa0f8
Nov 02 14:56:22.285499 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.285402    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-64bm4"]
Nov 02 14:56:22.302557 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:56:22.302514    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63faa28d_8806_4332_99c4_9f715a0b1a86.slice/crio-d2ae259552fb008b23239e86e9685ae682ff6eb2f0223a457ecfd672eb229d94 WatchSource:0}: Error finding container d2ae259552fb008b23239e86e9685ae682ff6eb2f0223a457ecfd672eb229d94: Status 404 returned error can't find the container with id d2ae259552fb008b23239e86e9685ae682ff6eb2f0223a457ecfd672eb229d94
Nov 02 14:56:22.318055 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.318027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.318164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.318231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.318257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.318286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.318363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318169    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318534    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls podName:7ff63dfc-17d4-4efc-a23a-d2e46a6885fc nodeName:}" failed. No retries permitted until 2026-11-02 14:56:23.318521983 +0000 UTC m=+33.813115884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vhx5m" (UID: "7ff63dfc-17d4-4efc-a23a-d2e46a6885fc") : secret "samples-operator-tls" not found
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318576    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318597    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls podName:885145c5-6077-4435-89ce-73af1e6a5600 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:23.318590613 +0000 UTC m=+33.813184513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-q2t8x" (UID: "885145c5-6077-4435-89ce-73af1e6a5600") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318629    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318646    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls podName:b639a9d2-f451-4cb1-a31f-c3de10b8d3c1 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:23.318639785 +0000 UTC m=+33.813233685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls") pod "dns-default-f6fgv" (UID: "b639a9d2-f451-4cb1-a31f-c3de10b8d3c1") : secret "dns-default-metrics-tls" not found
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318676    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318695    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert podName:fb39336d-67c0-4cf2-aaff-c873d361f9b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:23.318689127 +0000 UTC m=+33.813283026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mf5tv" (UID: "fb39336d-67c0-4cf2-aaff-c873d361f9b2") : secret "networking-console-plugin-cert" not found
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318173    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318706    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74fd4fbb69-thwc5: secret "image-registry-tls" not found
Nov 02 14:56:22.318769 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318725    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls podName:8d2ae30e-0191-41ff-85da-dedf17ad27b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:23.318719754 +0000 UTC m=+33.813313667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls") pod "image-registry-74fd4fbb69-thwc5" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2") : secret "image-registry-tls" not found
Nov 02 14:56:22.320743 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318493    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:22.320743 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318734    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcff7b7b8-bck5q: secret "image-registry-tls" not found
Nov 02 14:56:22.320743 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.318748    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls podName:fac285e0-f60a-4f53-a7e8-dee42bb83a0a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:23.318743844 +0000 UTC m=+33.813337744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls") pod "image-registry-dcff7b7b8-bck5q" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a") : secret "image-registry-tls" not found
Nov 02 14:56:22.349224 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.349195    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"]
Nov 02 14:56:22.364195 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.364174    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q"]
Nov 02 14:56:22.365079 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.365057    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"]
Nov 02 14:56:22.373614 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:56:22.373302    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0e9509_693a_4b99_b5c5_32770e758b5a.slice/crio-42dfad59a2b6d96988d657929565ea6e78badddfd418f4828bf3f0837ff6987f WatchSource:0}: Error finding container 42dfad59a2b6d96988d657929565ea6e78badddfd418f4828bf3f0837ff6987f: Status 404 returned error can't find the container with id 42dfad59a2b6d96988d657929565ea6e78badddfd418f4828bf3f0837ff6987f
Nov 02 14:56:22.373696 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:56:22.373646    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234a4a99_5491_44ef_b58c_3e4ab32584e5.slice/crio-04ee49d43983c5f89e295b788d4787753e97c452563b40872eb06125e8652928 WatchSource:0}: Error finding container 04ee49d43983c5f89e295b788d4787753e97c452563b40872eb06125e8652928: Status 404 returned error can't find the container with id 04ee49d43983c5f89e295b788d4787753e97c452563b40872eb06125e8652928
Nov 02 14:56:22.399377 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.397641    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l"]
Nov 02 14:56:22.402140 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:56:22.402113    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c6485e_f8bc_4b94_839c_746297629798.slice/crio-6b7bda4a5f49de4023ffa296cdac9f969e248682be6d98b2d87f89a1c974882b WatchSource:0}: Error finding container 6b7bda4a5f49de4023ffa296cdac9f969e248682be6d98b2d87f89a1c974882b: Status 404 returned error can't find the container with id 6b7bda4a5f49de4023ffa296cdac9f969e248682be6d98b2d87f89a1c974882b
Nov 02 14:56:22.406449 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.406429    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jcqwk"]
Nov 02 14:56:22.409538 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:56:22.409510    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb94c55_ff22_4e24_99da_047c765a8598.slice/crio-27c37d9ba6ac50422ef80c63a7ecf877ef12253b96aee46b68bbd1b3870fd57d WatchSource:0}: Error finding container 27c37d9ba6ac50422ef80c63a7ecf877ef12253b96aee46b68bbd1b3870fd57d: Status 404 returned error can't find the container with id 27c37d9ba6ac50422ef80c63a7ecf877ef12253b96aee46b68bbd1b3870fd57d
Nov 02 14:56:22.508589 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.508553    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v4f7p"]
Nov 02 14:56:22.512305 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:56:22.512282    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25b0184_4053_4694_acc1_c26e9f5602aa.slice/crio-ab7fbd6102be027382fccfa8315c9fdcd170c16952c835c1a7529b627fc887a1 WatchSource:0}: Error finding container ab7fbd6102be027382fccfa8315c9fdcd170c16952c835c1a7529b627fc887a1: Status 404 returned error can't find the container with id ab7fbd6102be027382fccfa8315c9fdcd170c16952c835c1a7529b627fc887a1
Nov 02 14:56:22.544545 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.542518    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"]
Nov 02 14:56:22.544722 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:56:22.544697    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc12a767_3259_4dc0_a809_d123b58334e3.slice/crio-ca013e6d5fafdd555a354b13627771bea71a3d124fa8f7e19b8058acea1a1c92 WatchSource:0}: Error finding container ca013e6d5fafdd555a354b13627771bea71a3d124fa8f7e19b8058acea1a1c92: Status 404 returned error can't find the container with id ca013e6d5fafdd555a354b13627771bea71a3d124fa8f7e19b8058acea1a1c92
Nov 02 14:56:22.621069 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.621039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:22.621180 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.621111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:22.621229 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.621215    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:56:22.621263 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.621221    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:24.621202432 +0000 UTC m=+35.115796350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : configmap references non-existent config key: service-ca.crt
Nov 02 14:56:22.621263 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:22.621255    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:24.621244354 +0000 UTC m=+35.115838253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : secret "router-metrics-certs-default" not found
Nov 02 14:56:22.623378 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:22.623355    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-ggtvw"]
Nov 02 14:56:22.626641 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:56:22.626615    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063ba34d_7544_410f_b861_5c300cc8f6d4.slice/crio-51b01d8c1cb6e8fc8997b625267adbda8c02207a0bb2cfa936d5d64ca7bbb6b8 WatchSource:0}: Error finding container 51b01d8c1cb6e8fc8997b625267adbda8c02207a0bb2cfa936d5d64ca7bbb6b8: Status 404 returned error can't find the container with id 51b01d8c1cb6e8fc8997b625267adbda8c02207a0bb2cfa936d5d64ca7bbb6b8
Nov 02 14:56:23.165071 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.165030    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv" event={"ID":"bf04dbe2-4354-4849-959c-33002c7ece68","Type":"ContainerStarted","Data":"8521a45378aa3339f8971a0af3256c2b4610e7f8d6c79a06be26359dc8f1cabc"}
Nov 02 14:56:23.166221 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.166198    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ggtvw" event={"ID":"063ba34d-7544-410f-b861-5c300cc8f6d4","Type":"ContainerStarted","Data":"51b01d8c1cb6e8fc8997b625267adbda8c02207a0bb2cfa936d5d64ca7bbb6b8"}
Nov 02 14:56:23.167879 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.167853    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-v4f7p" event={"ID":"f25b0184-4053-4694-acc1-c26e9f5602aa","Type":"ContainerStarted","Data":"c20bc2cb78a6a951bc4729b65a5532ac0454d8d1040c7666a3a91cdf6f1ca432"}
Nov 02 14:56:23.167993 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.167895    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-v4f7p" event={"ID":"f25b0184-4053-4694-acc1-c26e9f5602aa","Type":"ContainerStarted","Data":"ab7fbd6102be027382fccfa8315c9fdcd170c16952c835c1a7529b627fc887a1"}
Nov 02 14:56:23.169023 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.169000    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jcqwk" event={"ID":"9fb94c55-ff22-4e24-99da-047c765a8598","Type":"ContainerStarted","Data":"27c37d9ba6ac50422ef80c63a7ecf877ef12253b96aee46b68bbd1b3870fd57d"}
Nov 02 14:56:23.170116 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.170094    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d" event={"ID":"234a4a99-5491-44ef-b58c-3e4ab32584e5","Type":"ContainerStarted","Data":"04ee49d43983c5f89e295b788d4787753e97c452563b40872eb06125e8652928"}
Nov 02 14:56:23.171162 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.171142    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l" event={"ID":"27c6485e-f8bc-4b94-839c-746297629798","Type":"ContainerStarted","Data":"6b7bda4a5f49de4023ffa296cdac9f969e248682be6d98b2d87f89a1c974882b"}
Nov 02 14:56:23.172257 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.172239    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q" event={"ID":"be0e9509-693a-4b99-b5c5-32770e758b5a","Type":"ContainerStarted","Data":"42dfad59a2b6d96988d657929565ea6e78badddfd418f4828bf3f0837ff6987f"}
Nov 02 14:56:23.173632 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.173613    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xx5tn" event={"ID":"0290bc25-3e34-4d43-9215-f15cfc44f686","Type":"ContainerStarted","Data":"d57e874bd9a30d3e0cdd70d46b050f8d961c4c247dfc52509d1ac44fd3a20767"}
Nov 02 14:56:23.173700 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.173646    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xx5tn" event={"ID":"0290bc25-3e34-4d43-9215-f15cfc44f686","Type":"ContainerStarted","Data":"35549c1fd97c78be1f742ab6a916a99068c9e14728cf801854e852ccda8fa0f8"}
Nov 02 14:56:23.175110 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.175084    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-64bm4" event={"ID":"63faa28d-8806-4332-99c4-9f715a0b1a86","Type":"ContainerStarted","Data":"8dacbeb993294cc6376af14ead65d362dcb38f1fa32ee91e3eea3ea3c7487de2"}
Nov 02 14:56:23.175202 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.175130    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-64bm4" event={"ID":"63faa28d-8806-4332-99c4-9f715a0b1a86","Type":"ContainerStarted","Data":"d2ae259552fb008b23239e86e9685ae682ff6eb2f0223a457ecfd672eb229d94"}
Nov 02 14:56:23.176190 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.176171    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb" event={"ID":"fc12a767-3259-4dc0-a809-d123b58334e3","Type":"ContainerStarted","Data":"ca013e6d5fafdd555a354b13627771bea71a3d124fa8f7e19b8058acea1a1c92"}
Nov 02 14:56:23.328398 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.328357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:23.328579 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.328413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:23.328579 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.328458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:23.328579 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.328521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:23.328579 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.328526    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:23.328772 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.328577    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:56:23.328772 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.328598    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls podName:885145c5-6077-4435-89ce-73af1e6a5600 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:25.328578072 +0000 UTC m=+35.823171989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-q2t8x" (UID: "885145c5-6077-4435-89ce-73af1e6a5600") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:23.328772 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.328629    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls podName:b639a9d2-f451-4cb1-a31f-c3de10b8d3c1 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:25.328617355 +0000 UTC m=+35.823211255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls") pod "dns-default-f6fgv" (UID: "b639a9d2-f451-4cb1-a31f-c3de10b8d3c1") : secret "dns-default-metrics-tls" not found
Nov 02 14:56:23.328772 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.328654    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:23.328772 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.328666    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcff7b7b8-bck5q: secret "image-registry-tls" not found
Nov 02 14:56:23.328772 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.328705    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls podName:fac285e0-f60a-4f53-a7e8-dee42bb83a0a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:25.328691327 +0000 UTC m=+35.823285232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls") pod "image-registry-dcff7b7b8-bck5q" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a") : secret "image-registry-tls" not found
Nov 02 14:56:23.328772 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.328745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:23.329066 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.328790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:23.329066 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.328897    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:23.329066 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.328906    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74fd4fbb69-thwc5: secret "image-registry-tls" not found
Nov 02 14:56:23.329066 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.328937    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls podName:8d2ae30e-0191-41ff-85da-dedf17ad27b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:25.328926323 +0000 UTC m=+35.823520225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls") pod "image-registry-74fd4fbb69-thwc5" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2") : secret "image-registry-tls" not found
Nov 02 14:56:23.329066 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.328987    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:56:23.329066 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.329016    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert podName:fb39336d-67c0-4cf2-aaff-c873d361f9b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:25.329006957 +0000 UTC m=+35.823600861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mf5tv" (UID: "fb39336d-67c0-4cf2-aaff-c873d361f9b2") : secret "networking-console-plugin-cert" not found
Nov 02 14:56:23.329368 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.329287    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:56:23.329368 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.329351    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls podName:7ff63dfc-17d4-4efc-a23a-d2e46a6885fc nodeName:}" failed. No retries permitted until 2026-11-02 14:56:25.329340828 +0000 UTC m=+35.823934760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vhx5m" (UID: "7ff63dfc-17d4-4efc-a23a-d2e46a6885fc") : secret "samples-operator-tls" not found
Nov 02 14:56:23.733943 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.733557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:23.733943 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.733861    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 14:56:23.733943 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:23.733915    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs podName:9bea213e-5ff8-447d-98db-fa0ec51cda10 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:55.733898517 +0000 UTC m=+66.228492431 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs") pod "network-metrics-daemon-rkxt7" (UID: "9bea213e-5ff8-447d-98db-fa0ec51cda10") : secret "metrics-daemon-secret" not found
Nov 02 14:56:23.835807 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.835738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:23.853495 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.853465    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:23.937643 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.937002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw78t\" (UniqueName: \"kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t\") pod \"network-check-target-jjhkj\" (UID: \"8eaf7fe9-c300-4798-8212-e57e5a9cb03d\") " pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:23.940911 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.940859    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw78t\" (UniqueName: \"kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t\") pod \"network-check-target-jjhkj\" (UID: \"8eaf7fe9-c300-4798-8212-e57e5a9cb03d\") " pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:23.963324 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.963267    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xx5tn" podStartSLOduration=393.963254183 podStartE2EDuration="6m33.963254183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:23.962631372 +0000 UTC m=+34.457225291" watchObservedRunningTime="2026-11-02 14:56:23.963254183 +0000 UTC m=+34.457848102"
Nov 02 14:56:23.979488 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:23.979209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:24.081101 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:24.080829    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:24.084196 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:24.084005    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-64bm4" podStartSLOduration=394.083986924 podStartE2EDuration="6m34.083986924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:24.081211779 +0000 UTC m=+34.575805703" watchObservedRunningTime="2026-11-02 14:56:24.083986924 +0000 UTC m=+34.578580847"
Nov 02 14:56:24.382851 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:24.381550    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-v4f7p" podStartSLOduration=394.381528975 podStartE2EDuration="6m34.381528975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:24.232229962 +0000 UTC m=+34.726823910" watchObservedRunningTime="2026-11-02 14:56:24.381528975 +0000 UTC m=+34.876122898"
Nov 02 14:56:24.382851 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:24.382737    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jjhkj"]
Nov 02 14:56:24.389085 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:56:24.389059    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eaf7fe9_c300_4798_8212_e57e5a9cb03d.slice/crio-c91694f708270d0ad1b97681e97b0b5548021f657d3f0b42fcd1ff0a750b4dd4 WatchSource:0}: Error finding container c91694f708270d0ad1b97681e97b0b5548021f657d3f0b42fcd1ff0a750b4dd4: Status 404 returned error can't find the container with id c91694f708270d0ad1b97681e97b0b5548021f657d3f0b42fcd1ff0a750b4dd4
Nov 02 14:56:24.488225 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:24.488152    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nx6px"]
Nov 02 14:56:24.495955 ip-10-0-132-27 kubenswrapper[2577]: W1102 14:56:24.495930    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cff577_8a79_4b9e_89c8_a93e1ecf4ff0.slice/crio-6dd80de8fa69644e3697d5ebfb5c90eb265bbbeb9a89f8547e57a8425a083ad5 WatchSource:0}: Error finding container 6dd80de8fa69644e3697d5ebfb5c90eb265bbbeb9a89f8547e57a8425a083ad5: Status 404 returned error can't find the container with id 6dd80de8fa69644e3697d5ebfb5c90eb265bbbeb9a89f8547e57a8425a083ad5
Nov 02 14:56:24.645130 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:24.645048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:24.645391 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:24.645158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:24.645391 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:24.645372    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:56:24.645500 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:24.645429    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:28.645411009 +0000 UTC m=+39.140004932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : secret "router-metrics-certs-default" not found
Nov 02 14:56:24.645854 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:24.645829    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:28.645818197 +0000 UTC m=+39.140412110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : configmap references non-existent config key: service-ca.crt
Nov 02 14:56:25.184195 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:25.184093    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jjhkj" event={"ID":"8eaf7fe9-c300-4798-8212-e57e5a9cb03d","Type":"ContainerStarted","Data":"c91694f708270d0ad1b97681e97b0b5548021f657d3f0b42fcd1ff0a750b4dd4"}
Nov 02 14:56:25.186357 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:25.186269    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nx6px" event={"ID":"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0","Type":"ContainerStarted","Data":"6dd80de8fa69644e3697d5ebfb5c90eb265bbbeb9a89f8547e57a8425a083ad5"}
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:25.351793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:25.351857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:25.351911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:25.351941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:25.351984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:25.352029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352166    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352182    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcff7b7b8-bck5q: secret "image-registry-tls" not found
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352253    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls podName:fac285e0-f60a-4f53-a7e8-dee42bb83a0a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:29.352235242 +0000 UTC m=+39.846829156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls") pod "image-registry-dcff7b7b8-bck5q" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a") : secret "image-registry-tls" not found
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352343    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352389    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls podName:7ff63dfc-17d4-4efc-a23a-d2e46a6885fc nodeName:}" failed. No retries permitted until 2026-11-02 14:56:29.352367168 +0000 UTC m=+39.846961073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vhx5m" (UID: "7ff63dfc-17d4-4efc-a23a-d2e46a6885fc") : secret "samples-operator-tls" not found
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352450    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352461    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74fd4fbb69-thwc5: secret "image-registry-tls" not found
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352491    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls podName:8d2ae30e-0191-41ff-85da-dedf17ad27b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:29.352482088 +0000 UTC m=+39.847075990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls") pod "image-registry-74fd4fbb69-thwc5" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2") : secret "image-registry-tls" not found
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352550    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352580    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls podName:885145c5-6077-4435-89ce-73af1e6a5600 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:29.352571694 +0000 UTC m=+39.847165595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-q2t8x" (UID: "885145c5-6077-4435-89ce-73af1e6a5600") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:25.352788 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352634    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:56:25.353827 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352665    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls podName:b639a9d2-f451-4cb1-a31f-c3de10b8d3c1 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:29.35265383 +0000 UTC m=+39.847247730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls") pod "dns-default-f6fgv" (UID: "b639a9d2-f451-4cb1-a31f-c3de10b8d3c1") : secret "dns-default-metrics-tls" not found
Nov 02 14:56:25.353827 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352718    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:56:25.353827 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:25.352750    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert podName:fb39336d-67c0-4cf2-aaff-c873d361f9b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:29.352739937 +0000 UTC m=+39.847333850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mf5tv" (UID: "fb39336d-67c0-4cf2-aaff-c873d361f9b2") : secret "networking-console-plugin-cert" not found
Nov 02 14:56:26.194773 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:26.194736    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l" event={"ID":"27c6485e-f8bc-4b94-839c-746297629798","Type":"ContainerStarted","Data":"dc78edc88ddffce171a2adda7610226d8a2eedaabf817fb593c58b2cebfbbb7a"}
Nov 02 14:56:28.216380 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:28.215724    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/0.log"
Nov 02 14:56:28.216380 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:28.215764    2577 generic.go:356] "Generic (PLEG): container finished" podID="bf04dbe2-4354-4849-959c-33002c7ece68" containerID="2cf38e13333f40e302cc3a697efc8f060f194c3641ce5cc364c4418b591a200f" exitCode=255
Nov 02 14:56:28.216380 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:28.215837    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv" event={"ID":"bf04dbe2-4354-4849-959c-33002c7ece68","Type":"ContainerDied","Data":"2cf38e13333f40e302cc3a697efc8f060f194c3641ce5cc364c4418b591a200f"}
Nov 02 14:56:28.216380 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:28.216089    2577 scope.go:122] "RemoveContainer" containerID="2cf38e13333f40e302cc3a697efc8f060f194c3641ce5cc364c4418b591a200f"
Nov 02 14:56:28.225911 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:28.225323    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ggtvw" event={"ID":"063ba34d-7544-410f-b861-5c300cc8f6d4","Type":"ContainerStarted","Data":"1c2a61e5e15f57f689fafa9930029d09bc8798cf59e550b9151a9a4b40c497c5"}
Nov 02 14:56:28.228944 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:28.228447    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d" event={"ID":"234a4a99-5491-44ef-b58c-3e4ab32584e5","Type":"ContainerStarted","Data":"82f34400af35b3b9480695ff79e0ec70c83d42a7843156ed1c2ccf25cfcb6e6d"}
Nov 02 14:56:28.336481 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:28.336435    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l" podStartSLOduration=374.578730736 podStartE2EDuration="6m17.336417946s" totalImagesPullingTime="2.75768721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:26.3125757 +0000 UTC m=+36.807169623" watchObservedRunningTime="2026-11-02 14:56:28.336417946 +0000 UTC m=+38.831011872"
Nov 02 14:56:28.463776 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:28.461962    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-ggtvw" podStartSLOduration=373.141410878 podStartE2EDuration="6m18.461942851s" totalImagesPullingTime="5.320531973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:28.454407395 +0000 UTC m=+38.949001317" watchObservedRunningTime="2026-11-02 14:56:28.461942851 +0000 UTC m=+38.956536773"
Nov 02 14:56:28.571871 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:28.571782    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d" podStartSLOduration=382.010226976 podStartE2EDuration="6m27.571762863s" totalImagesPullingTime="5.561535887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:28.571410512 +0000 UTC m=+39.066004452" watchObservedRunningTime="2026-11-02 14:56:28.571762863 +0000 UTC m=+39.066356785"
Nov 02 14:56:28.689248 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:28.689205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:28.689447 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:28.689344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:28.689561 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:28.689541    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:56:28.689632 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:28.689615    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:36.689594712 +0000 UTC m=+47.184188628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : secret "router-metrics-certs-default" not found
Nov 02 14:56:28.690048 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:28.690025    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:36.690010085 +0000 UTC m=+47.184603998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : configmap references non-existent config key: service-ca.crt
Nov 02 14:56:29.233456 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.233425    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb" event={"ID":"fc12a767-3259-4dc0-a809-d123b58334e3","Type":"ContainerStarted","Data":"7afdbaf07390958b000f5acf887818ad28facc2349d602026d8d605d0bca32df"}
Nov 02 14:56:29.235783 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.235702    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/1.log"
Nov 02 14:56:29.237061 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.236162    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/0.log"
Nov 02 14:56:29.237061 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.236193    2577 generic.go:356] "Generic (PLEG): container finished" podID="bf04dbe2-4354-4849-959c-33002c7ece68" containerID="e65d58d2a1b20beec085715ef00a07f1555863222f9e3afe972c3aa920641f19" exitCode=255
Nov 02 14:56:29.237061 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.236378    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv" event={"ID":"bf04dbe2-4354-4849-959c-33002c7ece68","Type":"ContainerDied","Data":"e65d58d2a1b20beec085715ef00a07f1555863222f9e3afe972c3aa920641f19"}
Nov 02 14:56:29.237061 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.236427    2577 scope.go:122] "RemoveContainer" containerID="2cf38e13333f40e302cc3a697efc8f060f194c3641ce5cc364c4418b591a200f"
Nov 02 14:56:29.237061 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.236743    2577 scope.go:122] "RemoveContainer" containerID="e65d58d2a1b20beec085715ef00a07f1555863222f9e3afe972c3aa920641f19"
Nov 02 14:56:29.237061 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.236898    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rtsrv_openshift-console-operator(bf04dbe2-4354-4849-959c-33002c7ece68)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv" podUID="bf04dbe2-4354-4849-959c-33002c7ece68"
Nov 02 14:56:29.398282 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.398219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:29.399259 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.398461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:29.399259 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.398504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:29.399259 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.398571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:29.399259 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.398601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:29.399259 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.398701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:29.399259 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.398820    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:56:29.399259 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.398878    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert podName:fb39336d-67c0-4cf2-aaff-c873d361f9b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.398859494 +0000 UTC m=+47.893453412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mf5tv" (UID: "fb39336d-67c0-4cf2-aaff-c873d361f9b2") : secret "networking-console-plugin-cert" not found
Nov 02 14:56:29.399259 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.398877    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:29.399259 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.398892    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcff7b7b8-bck5q: secret "image-registry-tls" not found
Nov 02 14:56:29.399259 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.398924    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls podName:fac285e0-f60a-4f53-a7e8-dee42bb83a0a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.398912263 +0000 UTC m=+47.893506199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls") pod "image-registry-dcff7b7b8-bck5q" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a") : secret "image-registry-tls" not found
Nov 02 14:56:29.399259 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.399137    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:56:29.399259 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.399244    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls podName:b639a9d2-f451-4cb1-a31f-c3de10b8d3c1 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.399226305 +0000 UTC m=+47.893820206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls") pod "dns-default-f6fgv" (UID: "b639a9d2-f451-4cb1-a31f-c3de10b8d3c1") : secret "dns-default-metrics-tls" not found
Nov 02 14:56:29.400881 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.400857    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:29.401000 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.400887    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74fd4fbb69-thwc5: secret "image-registry-tls" not found
Nov 02 14:56:29.401000 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.400932    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls podName:8d2ae30e-0191-41ff-85da-dedf17ad27b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.400915404 +0000 UTC m=+47.895509310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls") pod "image-registry-74fd4fbb69-thwc5" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2") : secret "image-registry-tls" not found
Nov 02 14:56:29.401000 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.400994    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:56:29.401162 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.401025    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls podName:7ff63dfc-17d4-4efc-a23a-d2e46a6885fc nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.401015104 +0000 UTC m=+47.895609007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vhx5m" (UID: "7ff63dfc-17d4-4efc-a23a-d2e46a6885fc") : secret "samples-operator-tls" not found
Nov 02 14:56:29.401162 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.401121    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:29.401273 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.401174    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls podName:885145c5-6077-4435-89ce-73af1e6a5600 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.401157318 +0000 UTC m=+47.895751220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-q2t8x" (UID: "885145c5-6077-4435-89ce-73af1e6a5600") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:29.483950 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.483852    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb" podStartSLOduration=377.71702238 podStartE2EDuration="6m23.483832625s" totalImagesPullingTime="5.766810245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:29.36007655 +0000 UTC m=+39.854670473" watchObservedRunningTime="2026-11-02 14:56:29.483832625 +0000 UTC m=+39.978426548"
Nov 02 14:56:29.527045 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.527010    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dtxzd"]
Nov 02 14:56:29.549787 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.549760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.608612 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.608419    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 14:56:29.608612 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.608590    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 14:56:29.608828 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.608592    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fbnz4\""
Nov 02 14:56:29.615676 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.615567    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dtxzd"]
Nov 02 14:56:29.701617 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.701586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fb8\" (UniqueName: \"kubernetes.io/projected/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-api-access-s7fb8\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.701743 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.701697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.701816 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.701737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.701816 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.701796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-crio-socket\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.701920 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.701853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-data-volume\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.804134 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.803985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fb8\" (UniqueName: \"kubernetes.io/projected/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-api-access-s7fb8\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.805006 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.804461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.805006 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.804516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.805006 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.804572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-crio-socket\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.805006 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.804622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-data-volume\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.805006 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.804832    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:29.805006 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.804898    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:30.304879711 +0000 UTC m=+40.799473625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:29.805006 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.804954    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:56:29.805006 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.804965    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-data-volume\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.805006 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:29.804984    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:30.304974199 +0000 UTC m=+40.799568121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : configmap "kube-rbac-proxy" not found
Nov 02 14:56:29.811129 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.811100    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-crio-socket\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:29.898145 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:29.898109    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fb8\" (UniqueName: \"kubernetes.io/projected/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-api-access-s7fb8\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:30.241443 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:30.241348    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jcqwk" event={"ID":"9fb94c55-ff22-4e24-99da-047c765a8598","Type":"ContainerStarted","Data":"02de437079e0f9cb55802304b68eaf676ca8f2032cf80a19b982af63e217b6d0"}
Nov 02 14:56:30.243207 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:30.243166    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q" event={"ID":"be0e9509-693a-4b99-b5c5-32770e758b5a","Type":"ContainerStarted","Data":"6990d2d88bf3fef8234d8d3a825404cdbad09ec4a3dd817bf74084cb31b0917c"}
Nov 02 14:56:30.244794 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:30.244772    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/1.log"
Nov 02 14:56:30.245302 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:30.245283    2577 scope.go:122] "RemoveContainer" containerID="e65d58d2a1b20beec085715ef00a07f1555863222f9e3afe972c3aa920641f19"
Nov 02 14:56:30.245520 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:30.245494    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rtsrv_openshift-console-operator(bf04dbe2-4354-4849-959c-33002c7ece68)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv" podUID="bf04dbe2-4354-4849-959c-33002c7ece68"
Nov 02 14:56:30.313765 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:30.313396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:30.313765 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:30.313457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:30.313765 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:30.313548    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:56:30.313765 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:30.313683    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:31.313664578 +0000 UTC m=+41.808258483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : configmap "kube-rbac-proxy" not found
Nov 02 14:56:30.313765 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:30.313553    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:30.313765 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:30.313735    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:31.313724205 +0000 UTC m=+41.808318106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:30.359637 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:30.359573    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jcqwk" podStartSLOduration=3.595627564 podStartE2EDuration="10.359553598s" totalImagesPullingTime="6.763926034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:30.357980334 +0000 UTC m=+40.852574256" watchObservedRunningTime="2026-11-02 14:56:30.359553598 +0000 UTC m=+40.854147520"
Nov 02 14:56:30.479331 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:30.479266    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q" podStartSLOduration=363.195627725 podStartE2EDuration="6m10.479251701s" totalImagesPullingTime="7.283623976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:30.478880888 +0000 UTC m=+40.973474811" watchObservedRunningTime="2026-11-02 14:56:30.479251701 +0000 UTC m=+40.973845662"
Nov 02 14:56:31.249943 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:31.249879    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jjhkj" event={"ID":"8eaf7fe9-c300-4798-8212-e57e5a9cb03d","Type":"ContainerStarted","Data":"eccd16e65a37e5990cb0560f44998846f6dcac5ccc8eb74fa3a7327dedab2281"}
Nov 02 14:56:31.323254 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:31.323214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:31.323454 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:31.323269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:31.323454 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:31.323367    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:56:31.323454 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:31.323436    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:33.323415672 +0000 UTC m=+43.818009590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : configmap "kube-rbac-proxy" not found
Nov 02 14:56:31.323649 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:31.323593    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:31.323649 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:31.323634    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:33.323621733 +0000 UTC m=+43.818215638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:31.365820 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:31.365743    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jjhkj" podStartSLOduration=35.319233655 podStartE2EDuration="41.365725369s" totalImagesPullingTime="6.046491714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:31.365336705 +0000 UTC m=+41.859930621" watchObservedRunningTime="2026-11-02 14:56:31.365725369 +0000 UTC m=+41.860319294"
Nov 02 14:56:31.943710 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:31.943677    2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:31.943863 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:31.943736    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:31.944057 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:31.944041    2577 scope.go:122] "RemoveContainer" containerID="e65d58d2a1b20beec085715ef00a07f1555863222f9e3afe972c3aa920641f19"
Nov 02 14:56:31.944242 ip-10-0-132-27 kubenswrapper[2577]: E1102 14:56:31.944223    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rtsrv_openshift-console-operator(bf04dbe2-4354-4849-959c-33002c7ece68)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv" podUID="bf04dbe2-4354-4849-959c-33002c7ece68"
Nov 02 14:56:32.254636 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:32.254601    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nx6px" event={"ID":"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0","Type":"ContainerStarted","Data":"0bd3ca66cc7bf97c27c69d65e0bd0bdcb3c4339ac11a7f61e49844b1b403c41c"}
Nov 02 14:56:32.255025 ip-10-0-132-27 kubenswrapper[2577]: I1102 14:56:32.254794    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:32.322538 ip-10-0-132-27 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 14:56:32.332699 ip-10-0-132-27 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 14:56:32.332883 ip-10-0-132-27 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 14:56:32.332934 ip-10-0-132-27 systemd[1]: kubelet.service: Consumed 3.004s CPU time.
Nov 02 14:56:32.342962 ip-10-0-132-27 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 14:56:32.436720 ip-10-0-132-27 kubenswrapper[4891]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:56:32.436720 ip-10-0-132-27 kubenswrapper[4891]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 14:56:32.436720 ip-10-0-132-27 kubenswrapper[4891]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:56:32.436720 ip-10-0-132-27 kubenswrapper[4891]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:56:32.438727 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438642    4891 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:32.438727 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438728    4891 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438732    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438736    4891 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438738    4891 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438741    4891 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438744    4891 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438747    4891 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438750    4891 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438752    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438756    4891 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438759    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438761    4891 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438764    4891 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438766    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438769    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438772    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438776    4891 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438781    4891 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438785    4891 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:32.438804 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438788    4891 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438791    4891 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438795    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438797    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438808    4891 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438811    4891 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438814    4891 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438818    4891 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438821    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438824    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438828    4891 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438832    4891 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438835    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438838    4891 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438841    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438843    4891 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438846    4891 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438849    4891 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438851    4891 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438854    4891 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:32.439293 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438857    4891 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438860    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438862    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438865    4891 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438868    4891 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438870    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438873    4891 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438876    4891 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438878    4891 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438881    4891 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438884    4891 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438886    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438888    4891 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438891    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438896    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438899    4891 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438901    4891 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438904    4891 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438907    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:32.439826 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438910    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438912    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438915    4891 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438917    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438920    4891 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438923    4891 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438925    4891 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438928    4891 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438931    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438933    4891 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438936    4891 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438939    4891 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438941    4891 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438944    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438947    4891 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438950    4891 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438955    4891 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438957    4891 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438960    4891 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:32.440340 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438963    4891 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438965    4891 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438968    4891 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438970    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438973    4891 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438975    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438978    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438980    4891 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438984    4891 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438986    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438990    4891 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438993    4891 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438995    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.438998    4891 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439001    4891 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439003    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439006    4891 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439009    4891 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439011    4891 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439014    4891 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:32.440822 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439016    4891 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439019    4891 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439022    4891 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439024    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439027    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439029    4891 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439032    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439036    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439839    4891 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439847    4891 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439850    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439853    4891 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439855    4891 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439858    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439861    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439863    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439866    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439868    4891 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439871    4891 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439874    4891 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:32.441352 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439877    4891 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439879    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439882    4891 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439885    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439887    4891 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439890    4891 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439892    4891 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439895    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439898    4891 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439902    4891 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439904    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439906    4891 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439909    4891 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439912    4891 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439914    4891 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439917    4891 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439919    4891 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439922    4891 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439924    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439927    4891 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:32.441878 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439949    4891 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439953    4891 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439955    4891 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439962    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439965    4891 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439967    4891 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439970    4891 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439973    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439976    4891 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439978    4891 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439981    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439984    4891 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439986    4891 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439991    4891 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.439996    4891 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440000    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440003    4891 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440006    4891 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440009    4891 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:32.442397 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440012    4891 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440014    4891 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440017    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440020    4891 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440022    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440025    4891 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440027    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440030    4891 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440032    4891 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440035    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440038    4891 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440040    4891 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440043    4891 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440048    4891 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440057    4891 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440061    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440083    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440088    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440091    4891 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:32.442871 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440094    4891 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440097    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440100    4891 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440103    4891 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440105    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440108    4891 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440110    4891 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440113    4891 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440115    4891 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440118    4891 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440121    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440123    4891 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440126    4891 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440128    4891 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440131    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440133    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440136    4891 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440139    4891 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440141    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:32.443434 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440144    4891 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440146    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440149    4891 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440151    4891 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440154    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440156    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440161    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440164    4891 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440166    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440175    4891 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440178    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440180    4891 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440183    4891 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440185    4891 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440188    4891 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440192    4891 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440195    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440715    4891 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440722    4891 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440725    4891 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:32.443921 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440728    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440730    4891 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440734    4891 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440736    4891 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440739    4891 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440742    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440744    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440747    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440749    4891 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440753    4891 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440757    4891 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440760    4891 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440763    4891 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440766    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440770    4891 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440772    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440775    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440778    4891 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440783    4891 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:32.444443 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440786    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440789    4891 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440792    4891 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440795    4891 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440797    4891 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440800    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440802    4891 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440805    4891 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440808    4891 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440810    4891 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440813    4891 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440816    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440818    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440821    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440823    4891 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440826    4891 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440829    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440831    4891 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440834    4891 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440837    4891 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:32.444948 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440839    4891 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440842    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440844    4891 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440847    4891 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440850    4891 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440852    4891 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440854    4891 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440857    4891 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440860    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440862    4891 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440865    4891 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440869    4891 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440873    4891 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440877    4891 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440880    4891 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440883    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440886    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440888    4891 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440891    4891 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:32.445447 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440894    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440896    4891 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440899    4891 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440901    4891 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440905    4891 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440907    4891 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440910    4891 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440913    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440915    4891 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440918    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440920    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440923    4891 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440925    4891 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440928    4891 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440944    4891 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440948    4891 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440953    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440955    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440958    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:32.445938 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440961    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440963    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440966    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440969    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440971    4891 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440975    4891 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440978    4891 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440980    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440983    4891 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440985    4891 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440989    4891 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440991    4891 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440994    4891 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.440996    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441000    4891 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441003    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441005    4891 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441008    4891 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441010    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441013    4891 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:32.446435 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441015    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441018    4891 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441020    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441023    4891 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441026    4891 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441028    4891 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441112    4891 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441121    4891 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441127    4891 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441133    4891 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441137    4891 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441140    4891 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441145    4891 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441149    4891 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441152    4891 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441155    4891 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441159    4891 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441164    4891 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441167    4891 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441170    4891 flags.go:64] FLAG: --cgroup-root=""
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441174    4891 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441177    4891 flags.go:64] FLAG: --client-ca-file=""
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441180    4891 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441183    4891 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 14:56:32.446936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441188    4891 flags.go:64] FLAG: --cluster-domain=""
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441191    4891 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441194    4891 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441197    4891 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441201    4891 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441205    4891 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441208    4891 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441211    4891 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441214    4891 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441217    4891 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441220    4891 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441223    4891 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441226    4891 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441229    4891 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441234    4891 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441237    4891 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441240    4891 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441244    4891 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441248    4891 flags.go:64] FLAG: --enable-server="true"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441251    4891 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441255    4891 flags.go:64] FLAG: --event-burst="100"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441258    4891 flags.go:64] FLAG: --event-qps="50"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441261    4891 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441264    4891 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 14:56:32.447549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441267    4891 flags.go:64] FLAG: --eviction-hard=""
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441273    4891 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441276    4891 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441280    4891 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441283    4891 flags.go:64] FLAG: --eviction-soft=""
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441286    4891 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441289    4891 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441292    4891 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441296    4891 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441299    4891 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441302    4891 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441305    4891 flags.go:64] FLAG: --feature-gates=""
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441323    4891 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441326    4891 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441329    4891 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441332    4891 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441336    4891 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441339    4891 flags.go:64] FLAG: --help="false"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441342    4891 flags.go:64] FLAG: --hostname-override="ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441345    4891 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441348    4891 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441351    4891 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441355    4891 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 14:56:32.448178 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441359    4891 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441362    4891 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441367    4891 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441370    4891 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441373    4891 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441376    4891 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441379    4891 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441382    4891 flags.go:64] FLAG: --kube-reserved=""
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441385    4891 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441388    4891 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441393    4891 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441396    4891 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441399    4891 flags.go:64] FLAG: --lock-file=""
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441402    4891 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441405    4891 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441408    4891 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441414    4891 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441417    4891 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441420    4891 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441423    4891 flags.go:64] FLAG: --logging-format="text"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441426    4891 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441430    4891 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441433    4891 flags.go:64] FLAG: --manifest-url=""
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441436    4891 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441440    4891 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 14:56:32.448888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441444    4891 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441447    4891 flags.go:64] FLAG: --max-pods="110"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441450    4891 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441453    4891 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441456    4891 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441460    4891 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441463    4891 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441466    4891 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441469    4891 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441476    4891 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441479    4891 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441482    4891 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441485    4891 flags.go:64] FLAG: --pod-cidr=""
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441488    4891 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441491    4891 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441495    4891 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441498    4891 flags.go:64] FLAG: --port="10250"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441502    4891 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441505    4891 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0f690a810e9b77533"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441508    4891 flags.go:64] FLAG: --qos-reserved=""
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441511    4891 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441515    4891 flags.go:64] FLAG: --register-node="true"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441518    4891 flags.go:64] FLAG: --register-with-taints=""
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441521    4891 flags.go:64] FLAG: --registry-burst="10"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441524    4891 flags.go:64] FLAG: --registry-qps="5"
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441527    4891 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 14:56:32.449699 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441531    4891 flags.go:64] FLAG: --reserved-memory=""
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441535    4891 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441539    4891 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441542    4891 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441544    4891 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441547    4891 flags.go:64] FLAG: --runonce="false"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441550    4891 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441554    4891 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441556    4891 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441559    4891 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441562    4891 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441565    4891 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441569    4891 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441571    4891 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441574    4891 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441579    4891 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441582    4891 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441585    4891 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441588    4891 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441591    4891 flags.go:64] FLAG: --system-cgroups=""
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441594    4891 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441599    4891 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441602    4891 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441606    4891 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441611    4891 flags.go:64] FLAG: --tls-min-version=""
Nov 02 14:56:32.450606 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441614    4891 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441617    4891 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441620    4891 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441623    4891 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441626    4891 flags.go:64] FLAG: --v="2"
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441630    4891 flags.go:64] FLAG: --version="false"
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441634    4891 flags.go:64] FLAG: --vmodule=""
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441638    4891 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.441642    4891 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441771    4891 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441775    4891 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441777    4891 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441780    4891 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441783    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441785    4891 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441788    4891 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441791    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441794    4891 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441796    4891 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441799    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441801    4891 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441804    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:32.451576 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441812    4891 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441815    4891 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441818    4891 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441821    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441824    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441827    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441830    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441833    4891 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441838    4891 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441841    4891 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441843    4891 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441846    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441849    4891 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441852    4891 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441854    4891 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441857    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441862    4891 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441865    4891 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441868    4891 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:32.452131 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441872    4891 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441874    4891 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441877    4891 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441880    4891 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441883    4891 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441886    4891 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441889    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441892    4891 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441895    4891 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441898    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441901    4891 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441903    4891 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441906    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441911    4891 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441913    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441916    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441919    4891 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441922    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441925    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441927    4891 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:32.452630 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441930    4891 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441934    4891 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441937    4891 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441940    4891 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441943    4891 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441945    4891 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441948    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441951    4891 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441954    4891 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441957    4891 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441961    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441966    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441970    4891 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441974    4891 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441979    4891 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441983    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441987    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441991    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.441995    4891 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:32.453230 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442000    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442003    4891 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442006    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442009    4891 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442011    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442014    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442019    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442023    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442028    4891 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442031    4891 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442034    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442037    4891 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442039    4891 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442042    4891 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442047    4891 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442050    4891 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442052    4891 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442055    4891 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442058    4891 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:32.453821 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442061    4891 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442064    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442067    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442070    4891 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442072    4891 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442075    4891 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442077    4891 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442080    4891 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442083    4891 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442086    4891 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442089    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442092    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442096    4891 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442100    4891 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442103    4891 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442106    4891 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:32.454736 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.442135    4891 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442172    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442176    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442180    4891 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442184    4891 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442188    4891 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442192    4891 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442195    4891 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442198    4891 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442201    4891 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442204    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442208    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442212    4891 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442217    4891 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442219    4891 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442222    4891 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442225    4891 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442227    4891 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442230    4891 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442233    4891 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:32.455296 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442235    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442238    4891 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442240    4891 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442243    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442245    4891 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442249    4891 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442251    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442254    4891 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442257    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442259    4891 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442262    4891 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442264    4891 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442267    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442270    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442272    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442275    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442279    4891 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442282    4891 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442285    4891 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:32.456018 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442287    4891 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442290    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442292    4891 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442295    4891 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442299    4891 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442302    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442319    4891 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442323    4891 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442326    4891 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442329    4891 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442332    4891 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442335    4891 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442338    4891 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442341    4891 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442344    4891 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442347    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442351    4891 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442353    4891 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442356    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442359    4891 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:32.456598 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442362    4891 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442365    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442367    4891 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442370    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442373    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442375    4891 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442378    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442380    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442383    4891 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442388    4891 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442391    4891 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442394    4891 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442396    4891 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442399    4891 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442402    4891 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442404    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442411    4891 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442414    4891 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442416    4891 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442420    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:32.457089 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442424    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442428    4891 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442432    4891 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442436    4891 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442440    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442445    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442448    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442451    4891 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442453    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442456    4891 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442458    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442461    4891 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442465    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442468    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442470    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442473    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442476    4891 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442478    4891 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442481    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:32.457703 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442484    4891 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442486    4891 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442491    4891 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442494    4891 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442496    4891 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442499    4891 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442502    4891 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442507    4891 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.442511    4891 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.446280    4891 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.446295    4891 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.446323    4891 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.446328    4891 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446388    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446393    4891 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446396    4891 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446400    4891 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446403    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446406    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:32.458186 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446408    4891 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446411    4891 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446414    4891 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446417    4891 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446419    4891 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446422    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446426    4891 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446429    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446431    4891 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446434    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446437    4891 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446440    4891 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446443    4891 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446446    4891 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446449    4891 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446452    4891 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446455    4891 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446458    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446461    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:32.458674 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446464    4891 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446467    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446470    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446472    4891 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446475    4891 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446478    4891 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446481    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446484    4891 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446486    4891 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446489    4891 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446492    4891 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446494    4891 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446497    4891 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446500    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446503    4891 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446506    4891 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446508    4891 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446511    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446514    4891 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446517    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:32.459147 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446520    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446522    4891 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446525    4891 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446528    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446530    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446533    4891 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446535    4891 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446538    4891 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446541    4891 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446543    4891 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446546    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446549    4891 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446551    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446554    4891 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446557    4891 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446560    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446563    4891 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446565    4891 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446568    4891 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446570    4891 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:32.459679 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446573    4891 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446576    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446579    4891 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446581    4891 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446584    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446587    4891 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446590    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446594    4891 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446597    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446600    4891 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446603    4891 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446607    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446609    4891 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446612    4891 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446615    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446618    4891 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446622    4891 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446624    4891 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446627    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:32.460190 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446630    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446632    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446635    4891 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446638    4891 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446641    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446645    4891 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446648    4891 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446651    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446654    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446657    4891 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446660    4891 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446662    4891 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446665    4891 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446668    4891 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446670    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446673    4891 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446675    4891 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446678    4891 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446681    4891 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:32.460678 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446683    4891 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446686    4891 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446689    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.446716    4891 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446853    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446857    4891 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446860    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446863    4891 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446867    4891 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446870    4891 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446873    4891 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446875    4891 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446878    4891 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446881    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446884    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446886    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:32.461133 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446889    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446891    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446894    4891 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446897    4891 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446899    4891 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446902    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446904    4891 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446907    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446909    4891 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446912    4891 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446914    4891 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446917    4891 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446919    4891 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446922    4891 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446924    4891 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446927    4891 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446930    4891 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446932    4891 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446935    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446938    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:32.461600 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446941    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446943    4891 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446946    4891 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446949    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446952    4891 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446955    4891 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446958    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446960    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446964    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446966    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446969    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446972    4891 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446974    4891 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446976    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446979    4891 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446982    4891 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446984    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446987    4891 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446989    4891 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446992    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:32.462102 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446995    4891 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.446997    4891 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447000    4891 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447002    4891 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447005    4891 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447007    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447011    4891 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447014    4891 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447017    4891 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447020    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447023    4891 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447025    4891 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447028    4891 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447030    4891 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447033    4891 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447036    4891 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447039    4891 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447041    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447044    4891 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:32.462610 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447047    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447049    4891 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447052    4891 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447055    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447057    4891 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447060    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447064    4891 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447068    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447071    4891 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447074    4891 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447077    4891 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447079    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447081    4891 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447084    4891 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447086    4891 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447089    4891 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447091    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447094    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447096    4891 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:32.463071 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447099    4891 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447102    4891 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447105    4891 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447108    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447110    4891 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447113    4891 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447115    4891 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447118    4891 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447120    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447123    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447126    4891 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447128    4891 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447131    4891 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447134    4891 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447136    4891 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447139    4891 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:32.463550 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.447161    4891 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447194    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447198    4891 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447200    4891 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447204    4891 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447207    4891 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447209    4891 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447212    4891 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447215    4891 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447217    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447220    4891 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447222    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447225    4891 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447227    4891 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447230    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447233    4891 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447235    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447239    4891 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447243    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:32.464045 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447246    4891 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447249    4891 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447252    4891 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447254    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447257    4891 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447260    4891 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447262    4891 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447265    4891 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447267    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447270    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447273    4891 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447276    4891 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447279    4891 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447281    4891 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447284    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447286    4891 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447289    4891 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447292    4891 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447294    4891 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447296    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:32.464563 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447299    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447301    4891 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447324    4891 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447327    4891 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447330    4891 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447333    4891 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447335    4891 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447338    4891 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447341    4891 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447344    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447346    4891 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447349    4891 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447353    4891 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447355    4891 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447358    4891 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447360    4891 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447363    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447365    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447369    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:32.465051 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447371    4891 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447374    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447376    4891 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447379    4891 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447381    4891 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447385    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447387    4891 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447390    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447393    4891 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447396    4891 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447398    4891 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447401    4891 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447403    4891 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447406    4891 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447409    4891 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447413    4891 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447415    4891 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447418    4891 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447421    4891 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447423    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:32.465561 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447426    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447428    4891 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447431    4891 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447435    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447437    4891 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447440    4891 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447443    4891 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447445    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447448    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447451    4891 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447454    4891 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447457    4891 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447459    4891 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447461    4891 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447464    4891 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447467    4891 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447470    4891 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447472    4891 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447475    4891 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447478    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:32.466048 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447481    4891 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447483    4891 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447486    4891 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447489    4891 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447491    4891 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447494    4891 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447498    4891 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447503    4891 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:32.447505    4891 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.447555    4891 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.448624    4891 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.448670    4891 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.448857    4891 server.go:1014] "Starting client certificate rotation"
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.449012    4891 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.449082    4891 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:50:50 +0000 UTC" deadline="2028-07-20 19:27:41.70898894 +0000 UTC"
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.449123    4891 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15028h31m9.259869455s"
Nov 02 14:56:32.466556 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.449347    4891 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:56:32.466990 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.451813    4891 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:56:32.466990 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.453257    4891 log.go:25] "Validated CRI v1 runtime API"
Nov 02 14:56:32.466990 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.455239    4891 log.go:25] "Validated CRI v1 image API"
Nov 02 14:56:32.466990 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.455979    4891 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 14:56:32.466990 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.457818    4891 fs.go:135] Filesystem UUIDs: map[1ac83c1d-9ea1-408d-adec-86cd8565b53d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b2396700-4548-426b-b27a-196eecbb0b4f:/dev/nvme0n1p3]
Nov 02 14:56:32.467131 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.457839    4891 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04ee49d43983c5f89e295b788d4787753e97c452563b40872eb06125e8652928/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04ee49d43983c5f89e295b788d4787753e97c452563b40872eb06125e8652928/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09f5f8c60b67e971cecefc180ad34912a7c688b3b0d7a056d99b90caabd7703b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09f5f8c60b67e971cecefc180ad34912a7c688b3b0d7a056d99b90caabd7703b/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d45f714830b537a2feb50896809b1fe72666081846d09c5931670e07a683698/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d45f714830b537a2feb50896809b1fe72666081846d09c5931670e07a683698/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27c37d9ba6ac50422ef80c63a7ecf877ef12253b96aee46b68bbd1b3870fd57d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27c37d9ba6ac50422ef80c63a7ecf877ef12253b96aee46b68bbd1b3870fd57d/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35549c1fd97c78be1f742ab6a916a99068c9e14728cf801854e852ccda8fa0f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35549c1fd97c78be1f742ab6a916a99068c9e14728cf801854e852ccda8fa0f8/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/365b3df95f728f6953f143442eb4f2a78449283c2a46402d6fac2c87491832dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/365b3df95f728f6953f143442eb4f2a78449283c2a46402d6fac2c87491832dc/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42dfad59a2b6d96988d657929565ea6e78badddfd418f4828bf3f0837ff6987f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42dfad59a2b6d96988d657929565ea6e78badddfd418f4828bf3f0837ff6987f/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/451041e2651890f10aa467d5f33ade18f569e0bf4afcf49f25fe2f64da05ea15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/451041e2651890f10aa467d5f33ade18f569e0bf4afcf49f25fe2f64da05ea15/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51b01d8c1cb6e8fc8997b625267adbda8c02207a0bb2cfa936d5d64ca7bbb6b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51b01d8c1cb6e8fc8997b625267adbda8c02207a0bb2cfa936d5d64ca7bbb6b8/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/577421c14bf19c27ebec20ab6b2456e2f9f8b28f8e848db135575c07900c7cf8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/577421c14bf19c27ebec20ab6b2456e2f9f8b28f8e848db135575c07900c7cf8/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61642b8959214064dd84b6bb614a5e9d97a10e840e2dd10b2115977f7d660a2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61642b8959214064dd84b6bb614a5e9d97a10e840e2dd10b2115977f7d660a2c/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b7bda4a5f49de4023ffa296cdac9f969e248682be6d98b2d87f89a1c974882b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b7bda4a5f49de4023ffa296cdac9f969e248682be6d98b2d87f89a1c974882b/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6dd80de8fa69644e3697d5ebfb5c90eb265bbbeb9a89f8547e57a8425a083ad5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6dd80de8fa69644e3697d5ebfb5c90eb265bbbeb9a89f8547e57a8425a083ad5/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8521a45378aa3339f8971a0af3256c2b4610e7f8d6c79a06be26359dc8f1cabc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8521a45378aa3339f8971a0af3256c2b4610e7f8d6c79a06be26359dc8f1cabc/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a07090f931f3077f7a532bbdd8e28e8c071f4f7870bcecd391fabb7ea9cdbc0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a07090f931f3077f7a532bbdd8e28e8c071f4f7870bcecd391fabb7ea9cdbc0/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab7fbd6102be027382fccfa8315c9fdcd170c16952c835c1a7529b627fc887a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab7fbd6102be027382fccfa8315c9fdcd170c16952c835c1a7529b627fc887a1/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b993e107d2170bf2292fd26237cb9e2465fdef5970b188455b859a69283e5ece/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b993e107d2170bf2292fd26237cb9e2465fdef5970b188455b859a69283e5ece/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c91694f708270d0ad1b97681e97b0b5548021f657d3f0b42fcd1ff0a750b4dd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c91694f708270d0ad1b97681e97b0b5548021f657d3f0b42fcd1ff0a750b4dd4/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca013e6d5fafdd555a354b13627771bea71a3d124fa8f7e19b8058acea1a1c92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca013e6d5fafdd555a354b13627771bea71a3d124fa8f7e19b8058acea1a1c92/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2ae259552fb008b23239e86e9685ae682ff6eb2f0223a457ecfd672eb229d94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2ae259552fb008b23239e86e9685ae682ff6eb2f0223a457ecfd672eb229d94/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc884cd696d8fd330a276edb5d95785996d300ff3d789a0024c43253e48a12e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc884cd696d8fd330a276edb5d95785996d300ff3d789a0024c43253e48a12e4/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8d79991e743cf3435c16fdb7859603a3e1eec5bad724f20f1ea2a923cd7e6ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8d79991e743cf3435c16fdb7859603a3e1eec5bad724f20f1ea2a923cd7e6ba/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0290bc25-3e34-4d43-9215-f15cfc44f686/volumes/kubernetes.io~projected/kube-api-access-flmc9:{mountpoint:/var/lib/kubelet/pods/0290bc25-3e34-4d43-9215-f15cfc44f686/volumes/kubernetes.io~projected/kube-api-access-flmc9 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/063ba34d-7544-410f-b861-5c300cc8f6d4/volumes/kubernetes.io~projected/kube-api-access-ftvbk:{mountpoint:/var/lib/kubelet/pods/063ba34d-7544-410f-b861-5c300cc8f6d4/volumes/kubernetes.io~projected/kube-api-access-ftvbk major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/063ba34d-7544-410f-b861-5c300cc8f6d4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/063ba34d-7544-410f-b861-5c300cc8f6d4/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/100e3924-bcfc-4208-84ea-c1ff1656e949/volumes/kubernetes.io~projected/kube-api-access-npd9z:{mountpoint:/var/lib/kubelet/pods/100e3924-bcfc-4208-84ea-c1ff1656e949/volumes/kubernetes.io~projected/kube-api-access-npd9z major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19ce39d1-e591-4af6-81e4-63d7d7b255c4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/19ce39d1-e591-4af6-81e4-63d7d7b255c4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19ce39d1-e591-4af6-81e4-63d7d7b255c4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/19ce39d1-e591-4af6-81e4-63d7d7b255c4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19ce39d1-e591-4af6-81e4-63d7d7b255c4/volumes/kubernetes.io~projected/kube-api-access-dsk26:{mountpoint:/var/lib/kubelet/pods/19ce39d1-e591-4af6-81e4-63d7d7b255c4/volumes/kubernetes.io~projected/kube-api-access-dsk26 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bcba632-1434-4533-951f-84b8bba1f0d8/volumes/kubernetes.io~projected/kube-api-access-psj4b:{mountpoint:/var/lib/kubelet/pods/1bcba632-1434-4533-951f-84b8bba1f0d8/volumes/kubernetes.io~projected/kube-api-access-psj4b major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bcba632-1434-4533-951f-84b8bba1f0d8/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/1bcba632-1434-4533-951f-84b8bba1f0d8/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bcba632-1434-4533-951f-84b8bba1f0d8/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/1bcba632-1434-4533-951f-84b8bba1f0d8/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/234a4a99-5491-44ef-b58c-3e4ab32584e5/volumes/kubernetes.io~projected/kube-api-access-2znqw:{mountpoint:/var/lib/kubelet/pods/234a4a99-5491-44ef-b58c-3e4ab32584e5/volumes/kubernetes.io~projected/kube-api-access-2znqw major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/234a4a99-5491-44ef-b58c-3e4ab32584e5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/234a4a99-5491-44ef-b58c-3e4ab32584e5/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27c6485e-f8bc-4b94-839c-746297629798/volumes/kubernetes.io~projected/kube-api-access-87hwb:{mountpoint:/var/lib/kubelet/pods/27c6485e-f8bc-4b94-839c-746297629798/volumes/kubernetes.io~projected/kube-api-access-87hwb major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac/volumes/kubernetes.io~projected/kube-api-access-b8n6k:{mountpoint:/var/lib/kubelet/pods/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac/volumes/kubernetes.io~projected/kube-api-access-b8n6k major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49e40d7f-ab6a-4817-ba3e-3e2ab361b454/volumes/kubernetes.io~projected/kube-api-access-k2pbg:{mountpoint:/var/lib/kubelet/pods/49e40d7f-ab6a-4817-ba3e-3e2ab361b454/volumes/kubernetes.io~projected/kube-api-access-k2pbg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63faa28d-8806-4332-99c4-9f715a0b1a86/volumes/kubernetes.io~projected/kube-api-access-kg79x:{mountpoint:/var/lib/kubelet/pods/63faa28d-8806-4332-99c4-9f715a0b1a86/volumes/kubernetes.io~projected/kube-api-access-kg79x major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f2ebec5-9906-438f-b2ce-010345fa49ba/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6f2ebec5-9906-438f-b2ce-010345fa49ba/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f2ebec5-9906-438f-b2ce-010345fa49ba/volumes/kubernetes.io~projected/kube-api-access-rmfrd:{mountpoint:/var/lib/kubelet/pods/6f2ebec5-9906-438f-b2ce-010345fa49ba/volumes/kubernetes.io~projected/kube-api-access-rmfrd major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f2ebec5-9906-438f-b2ce-010345fa49ba/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6f2ebec5-9906-438f-b2ce-010345fa49ba/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74049a08-892a-4ae2-a5a5-161da6182c1c/volumes/kubernetes.io~projected/kube-api-access-96w6k:{mountpoint:/var/lib/kubelet/pods/74049a08-892a-4ae2-a5a5-161da6182c1c/volumes/kubernetes.io~projected/kube-api-access-96w6k major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc/volumes/kubernetes.io~projected/kube-api-access-z9tf9:{mountpoint:/var/lib/kubelet/pods/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc/volumes/kubernetes.io~projected/kube-api-access-z9tf9 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/885145c5-6077-4435-89ce-73af1e6a5600/volumes/kubernetes.io~projected/kube-api-access-cshc2:{mountpoint:/var/lib/kubelet/pods/885145c5-6077-4435-89ce-73af1e6a5600/volumes/kubernetes.io~projected/kube-api-access-cshc2 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes/kubernetes.io~projected/kube-api-access-2fwhf:{mountpoint:/var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes/kubernetes.io~projected/kube-api-access-2fwhf major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8eaf7fe9-c300-4798-8212-e57e5a9cb03d/volumes/kubernetes.io~projected/kube-api-access-hw78t:{mountpoint:/var/lib/kubelet/pods/8eaf7fe9-c300-4798-8212-e57e5a9cb03d/volumes/kubernetes.io~projected/kube-api-access-hw78t major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bb718bc-429a-4875-bd8d-d52dbba0b252/volumes/kubernetes.io~projected/kube-api-access-vzqh5:{mountpoint:/var/lib/kubelet/pods/9bb718bc-429a-4875-bd8d-d52dbba0b252/volumes/kubernetes.io~projected/kube-api-access-vzqh5 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bea213e-5ff8-447d-98db-fa0ec51cda10/volumes/kubernetes.io~projected/kube-api-access-bx97c:{mountpoint:/var/lib/kubelet/pods/9bea213e-5ff8-447d-98db-fa0ec51cda10/volumes/kubernetes.io~projected/kube-api-access-bx97c major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fb94c55-ff22-4e24-99da-047c765a8598/volumes/kubernetes.io~projected/kube-api-access-xvkkc:{mountpoint:/var/lib/kubelet/pods/9fb94c55-ff22-4e24-99da-047c765a8598/volumes/kubernetes.io~projected/kube-api-access-xvkkc major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fb94c55-ff22-4e24-99da-047c765a8598/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9fb94c55-ff22-4e24-99da-047c765a8598/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acede18e-4d8e-47ee-b31f-918658a9ffba/volumes/kubernetes.io~projected/kube-api-access-7n9ql:{mountpoint:/var/lib/kubelet/pods/acede18e-4d8e-47ee-b31f-918658a9ffba/volumes/kubernetes.io~projected/kube-api-access-7n9ql major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1/volumes/kubernetes.io~projected/kube-api-access-dtqv7:{mountpoint:/var/lib/kubelet/pods/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1/volumes/kubernetes.io~projected/kube-api-access-dtqv7 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63/volumes/kubernetes.io~projected/kube-api-access-s7fb8:{mountpoint:/var/lib/kubelet/pods/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63/volumes/kubernetes.io~projected/kube-api-access-s7fb8 major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be0e9509-693a-4b99-b5c5-32770e758b5a/volumes/kubernetes.io~projected/kube-api-access-64mls:{mountpoint:/var/lib/kubelet/pods/be0e9509-693a-4b99-b5c5-32770e758b5a/volumes/kubernetes.io~projected/kube-api-access-64mls major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf04dbe2-4354-4849-959c-33002c7ece68/volumes/kubernetes.io~projected/kube-api-access-9l647:{mountpoint:/var/lib/kubelet/pods/bf04dbe2-4354-4849-959c-33002c7ece68/volumes/kubernetes.io~projected/kube-api-access-9l647 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf04dbe2-4354-4849-959c-33002c7ece68/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bf04dbe2-4354-4849-959c-33002c7ece68/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7d93f72-6fc0-469a-8c16-9feed09bda86/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c7d93f72-6fc0-469a-8c16-9feed09bda86/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f25b0184-4053-4694-acc1-c26e9f5602aa/volumes/kubernetes.io~projected/kube-api-access-khdhh:{mountpoint:/var/lib/kubelet/pods/f25b0184-4053-4694-acc1-c26e9f5602aa/volumes/kubernetes.io~projected/kube-api-access-khdhh major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes/kubernetes.io~projected/kube-api-access-qbskj:{mountpoint:/var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes/kubernetes.io~projected/kube-api-access-qbskj major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc12a767-3259-4dc0-a809-d123b58334e3/volumes/kubernetes.io~projected/kube-api-access-klllg:{mountpoint:/var/lib/kubelet/pods/fc12a767-3259-4dc0-a809-d123b58334e3/volumes/kubernetes.io~projected/kube-api-access-klllg major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc12a767-3259-4dc0-a809-d123b58334e3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fc12a767-3259-4dc0-a809-d123b58334e3/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5414255f775911f791f5adde739758090815de3863268f38db980c04a7352f02/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/dab33f6cd7885c8d8080f856db9ffa508380028b71e0132605f8ab69c12b4e70/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d5084ef39bf992bb47aef13be11331f6a701e27790b4e91c16ea1d1beb0e056e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/1ecf6848fea5b82943519ee573f2aec82a0008364f255e34523f01ccea434869/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/811e72aa42f5d3ece1d0f7d5f184d0d3797af560e4d38ac3a95b11140b3eade0/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4acdd501be94637500f63017962086d5218180516b077b5abaf5bc21fdc226cd/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9ae5b25587c4a97ed9e836ab69651ced2028e03eb7caacf25b99a525ab23cca6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/66033834e791a711d3147ae4423abb3ee2898ae2bfdb97584d04ae904fa703cd/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2f6ff9b7da3da42c089fb45b1ce7047b4c5e1571451f75aaa67263ae529f6ee6/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/def157f801b55b9d56d69a0b4846f0fcd233ec71ab2f25a56814e423c47c6465/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/32752fbf30241cd9d99e7b166c54a828f9dae912d39c04689a2df2d2a088c8c2/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/5fc686f404dc5c19192e28c3c68779d499287d07e18434e38f7f4a6ee5b88ec0/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/092e99c27cfa4bdda39f1de7532df43bd5113a25d6f1dd4e1cf3acffcc8234f7/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/daab47cb777f56d442c86edfb9a0cdaec8440782c3df3b9aa8c6994f38e0a5c9/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0c1ed06ddf6f580e299af940526eec057eafcd495810bac81d410947cf6c96c3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9a58854d90b14ca564271a3b5604d59eaa3a7245ff8d8805525739561f013076/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c1d2fbe069f3b9476b16eaf14d9a47505a4c2b8290805411bda56e1331056562/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d1adaad8f7a0e8fadc3e87bd6113cfd63e99231b1b442c4877de62dc0cd10984/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/05765182401c8818e73328ebc83dabdd73c7b58078210765951f089da6660dfa/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/82bed345230e9f5fccfee591c3e2afddd87d65f6d86e620736317de9de5162a0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6f1f2ef6d7b8ca7c7ca5b44dd93774c8cb009c37777ca18a5326693546f11642/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5812a2daae2a621acb34699fd76cb5eaddf1f920e218c2246612e8f8ef67c2fe/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/9ba41847b433c2ffb97d3d5044bfd759a0327238f5a9db0d844a997531317b5c/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/d83df91da26e25d1d601488aff59c91ac7a480969221d585bc15a33cfc06d61d/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/ff379ae43867798382fc4ceeb5034a9d3721375bfbf5ccc34c19c45c519bc24d/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/3f03e8ce8a36f7919d2ee42cf047dd96358893406ad730efd2143e71683c9dcf/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/bd6e0dd3a75b2cc9e19af4faf95ada20f5a58061e65640d2cad02b7d0d104cf3/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/392eb5efd2d8de9fd371d68ce5a595f98ca6645c0ad32182375a0b9e8da0c1a4/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/41a370ffd1006124ca0b941b7ff492e1cf7ceb22b9ec8610bdc4a507bbe45d98/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/dd8a2deb68b0fd4b16c60b9c3ba3a133a571c5aec0f1011d19c225cad91b86a5/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/5752741175cb32f15a05725afe6bb3f5b4bd846a65ab118f0ec5e89e4bb25672/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/0af8abb0b0681399f78e9d42389196a8bde0f819e4015e8b592454b38e110019/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/ae213c1f4d88c08c0a09c39c4ec6be3a4199d5b3b80a6aca59a3fb675a8749e1/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/ccbef395d43d34abb7cbc72e85c8c97b6c75b93861257a70baaed6558415f356/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/691d033b582e2ef58faa837c58c80459b035f8b2caf28250236b3bc35d393a27/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/cd01c02b8e8f7778fd86b4e7e79b0cf10b0b931518ea96246d8ea549098f1cf8/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/0fd59bdb6d56704cf61b73358f413b69ae7a2592cd3510195742bb0673bce688/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/fcdd00a5b35a0275128faf021a99dfd42dfb66b73cf6776070fef3c1e9b00634/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/5841a523ad9897fada744f7e4c519633475d6272432ec036ddf863b71b3df21b/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/bbaf32eabe1162cfbea5bee283b471428ad30db453ab27a6d1191470c094f186/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/db81723c60b4ad2761318c5f0aced5323e4a5da62a7551e5cf0ae8ce8546ca00/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/7382dd3ed8ad701fb7b7ebae5858b2affb6aaf654e57ddba3c4c58846c7b59b2/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/257b947251c4b269c472ab95626c27d7e767be729d5ce7b562367fe475efec75/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/9f29e02ec06b3003780dcf9c62932f006f0f606bdc52360b40eaaa88029deec6/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/204e7b6d39c1e77d72e6128d6b0bb45e91e886889d41e9e5aa17be5cd1ad5993/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/99a03b423f32f340f357118a297a8583db89c1d51ec1f0e8a01d80b0d305ab64/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/3faa6dd77b2cfafef5eda8ba04312fd51450435347c7f24bfd7fad07f397b505/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/fb110cd75ba5bb8ad0a61ef22a6c6b5d0f5129ec703f0a6e23362f0efc5016b0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/feb74444fe136cccfce9215c89adb89d12bfe642e6b91ccd8581987903d3ab6e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/6f8e48d2dfdf4bbf67e906e15d738166756efe98682693d049a93f8bc922f3ae/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/7401d928674686d44972f6cf0306d68f95f9140ccd5ba88c0293749b1ac0ff3e/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/6fd2f959bbaacd45c4ffe5ddffe750a1fb5dcef61d177b7c0c3618a0d6db6e99/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/e018bff296cb03187a84d489cde3dbbf6b7ebc94510fc0d916b0271e882fd0e9/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5b2fa9878f47f7d88d61f4eae87425e39b5b2439fcad3ec58a726979a7372b7d/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.466036    4891 manager.go:217] Machine: {Timestamp:2026-11-02 14:56:32.465585581 +0000 UTC m=+0.058202377 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099856 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fc4aea2d11a000b4dd42cf0126e38 SystemUUID:ec2fc4ae-a2d1-1a00-0b4d-d42cf0126e38 BootID:603f1e91-e129-4a21-b778-42331279ed2c Filesystems:[{Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0290bc25-3e34-4d43-9215-f15cfc44f686/volumes/kubernetes.io~projected/kube-api-access-flmc9 DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8eaf7fe9-c300-4798-8212-e57e5a9cb03d/volumes/kubernetes.io~projected/kube-api-access-hw78t DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2ae259552fb008b23239e86e9685ae682ff6eb2f0223a457ecfd672eb229d94/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc12a767-3259-4dc0-a809-d123b58334e3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63faa28d-8806-4332-99c4-9f715a0b1a86/volumes/kubernetes.io~projected/kube-api-access-kg79x DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6dd80de8fa69644e3697d5ebfb5c90eb265bbbeb9a89f8547e57a8425a083ad5/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19ce39d1-e591-4af6-81e4-63d7d7b255c4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b7bda4a5f49de4023ffa296cdac9f969e248682be6d98b2d87f89a1c974882b/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27c37d9ba6ac50422ef80c63a7ecf877ef12253b96aee46b68bbd1b3870fd57d/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b993e107d2170bf2292fd26237cb9e2465fdef5970b188455b859a69283e5ece/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6f2ebec5-9906-438f-b2ce-010345fa49ba/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8521a45378aa3339f8971a0af3256c2b4610e7f8d6c79a06be26359dc8f1cabc/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63/volumes/kubernetes.io~projected/kube-api-access-s7fb8 DeviceMajor:0 DeviceMinor:328 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes/kubernetes.io~projected/kube-api-access-qbskj DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc12a767-3259-4dc0-a809-d123b58334e3/volumes/kubernetes.io~projected/kube-api-access-klllg DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d45f714830b537a2feb50896809b1fe72666081846d09c5931670e07a683698/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/234a4a99-5491-44ef-b58c-3e4ab32584e5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/27c6485e-f8bc-4b94-839c-746297629798/volumes/kubernetes.io~projected/kube-api-access-87hwb DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1/volumes/kubernetes.io~projected/kube-api-access-dtqv7 DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab7fbd6102be027382fccfa8315c9fdcd170c16952c835c1a7529b627fc887a1/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c91694f708270d0ad1b97681e97b0b5548021f657d3f0b42fcd1ff0a750b4dd4/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes/kubernetes.io~projected/kube-api-access-2fwhf DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac/volumes/kubernetes.io~projected/kube-api-access-b8n6k DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9fb94c55-ff22-4e24-99da-047c765a8598/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/063ba34d-7544-410f-b861-5c300cc8f6d4/volumes/kubernetes.io~projected/kube-api-access-ftvbk DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9fb94c55-ff22-4e24-99da-047c765a8598/volumes/kubernetes.io~projected/kube-api-access-xvkkc DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be0e9509-693a-4b99-b5c5-32770e758b5a/volumes/kubernetes.io~projected/kube-api-access-64mls DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bb718bc-429a-4875-bd8d-d52dbba0b252/volumes/kubernetes.io~projected/kube-api-access-vzqh5 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bea213e-5ff8-447d-98db-fa0ec51cda10/volumes/kubernetes.io~projected/kube-api-access-bx97c DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09f5f8c60b67e971cecefc180ad34912a7c688b3b0d7a056d99b90caabd7703b/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/100e3924-bcfc-4208-84ea-c1ff1656e949/volumes/kubernetes.io~projected/kube-api-access-npd9z DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/365b3df95f728f6953f143442eb4f2a78449283c2a46402d6fac2c87491832dc/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f25b0184-4053-4694-acc1-c26e9f5602aa/volumes/kubernetes.io~projected/kube-api-access-khdhh DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51b01d8c1cb6e8fc8997b625267adbda8c02207a0bb2cfa936d5d64ca7bbb6b8/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/451041e2651890f10aa467d5f33ade18f569e0bf4afcf49f25fe2f64da05ea15/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1bcba632-1434-4533-951f-84b8bba1f0d8/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f2ebec5-9906-438f-b2ce-010345fa49ba/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/885145c5-6077-4435-89ce-73af1e6a5600/volumes/kubernetes.io~projected/kube-api-access-cshc2 DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61642b8959214064dd84b6bb614a5e9d97a10e840e2dd10b2115977f7d660a2c/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19ce39d1-e591-4af6-81e4-63d7d7b255c4/volumes/kubernetes.io~projected/kube-api-access-dsk26 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/acede18e-4d8e-47ee-b31f-918658a9ffba/volumes/kubernetes.io~projected/kube-api-access-7n9ql DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19ce39d1-e591-4af6-81e4-63d7d7b255c4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf04dbe2-4354-4849-959c-33002c7ece68/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf04dbe2-4354-4849-959c-33002c7ece68/volumes/kubernetes.io~projected/kube-api-access-9l647 DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a07090f931f3077f7a532bbdd8e28e8c071f4f7870bcecd391fabb7ea9cdbc0/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8d79991e743cf3435c16fdb7859603a3e1eec5bad724f20f1ea2a923cd7e6ba/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/577421c14bf19c27ebec20ab6b2456e2f9f8b28f8e848db135575c07900c7cf8/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc884cd696d8fd330a276edb5d95785996d300ff3d789a0024c43253e48a12e4/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1bcba632-1434-4533-951f-84b8bba1f0d8/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c7d93f72-6fc0-469a-8c16-9feed09bda86/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6f2ebec5-9906-438f-b2ce-010345fa49ba/volumes/kubernetes.io~projected/kube-api-access-rmfrd DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/063ba34d-7544-410f-b861-5c300cc8f6d4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1bcba632-1434-4533-951f-84b8bba1f0d8/volumes/kubernetes.io~projected/kube-api-access-psj4b DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35549c1fd97c78be1f742ab6a916a99068c9e14728cf801854e852ccda8fa0f8/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca013e6d5fafdd555a354b13627771bea71a3d124fa8f7e19b8058acea1a1c92/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42dfad59a2b6d96988d657929565ea6e78badddfd418f4828bf3f0837ff6987f/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04ee49d43983c5f89e295b788d4787753e97c452563b40872eb06125e8652928/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/234a4a99-5491-44ef-b58c-3e4ab32584e5/volumes/kubernetes.io~projected/kube-api-access-2znqw DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/49e40d7f-ab6a-4817-ba3e-3e2ab361b454/volumes/kubernetes.io~projected/kube-api-access-k2pbg DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc/volumes/kubernetes.io~projected/kube-api-access-z9tf9 DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74049a08-892a-4ae2-a5a5-161da6182c1c/volumes/kubernetes.io~projected/kube-api-access-96w6k DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:04ee49d43983c5f MacAddress:be:50:70:59:36:8e Speed:10000 Mtu:8901} {Name:27c37d9ba6ac504 MacAddress:7a:bc:85:23:98:02 Speed:10000 Mtu:8901} {Name:35549c1fd97c78b MacAddress:92:aa:92:21:b4:04 Speed:10000 Mtu:8901} {Name:42dfad59a2b6d96 MacAddress:02:33:51:a4:b0:1d Speed:10000 Mtu:8901} {Name:51b01d8c1cb6e8f MacAddress:d2:e8:6a:81:36:80 Speed:10000 Mtu:8901} {Name:6b7bda4a5f49de4 MacAddress:16:4e:da:21:05:27 Speed:10000 Mtu:8901} {Name:6dd80de8fa69644 MacAddress:c6:4c:78:70:82:41 Speed:10000 Mtu:8901} {Name:8521a45378aa333 MacAddress:ae:ce:ab:8d:3b:86 Speed:10000 Mtu:8901} {Name:ab7fbd6102be027 MacAddress:d6:b5:06:ef:35:46 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:13:1f:9f:0a:2d Speed:0 Mtu:9001} {Name:br-int MacAddress:6e:87:08:45:3d:fe Speed:0 Mtu:8901} {Name:c91694f708270d0 MacAddress:ea:6e:33:aa:a3:ed Speed:10000 Mtu:8901} {Name:ca013e6d5fafdd5 MacAddress:72:da:5a:0b:3c:ee Speed:10000 Mtu:8901} {Name:d2ae259552fb008 MacAddress:f6:66:12:ca:4b:e6 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:13:1f:9f:0a:2d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b2:e1:e7:56:1c:ad Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e2:af:77:49:6c:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.466379    4891 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.466423    4891 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.466699    4891 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.466716    4891 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-27.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.466842    4891 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.466848    4891 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.466861    4891 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.466881    4891 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.466899    4891 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.467058    4891 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.467180    4891 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.467215    4891 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.467227    4891 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.467237    4891 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.467246    4891 kubelet.go:405] "Adding apiserver pod source"
Nov 02 14:56:32.467382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.467254    4891 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 14:56:32.469972 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.469009    4891 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 14:56:32.469972 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.469029    4891 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:56:32.470138 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.470105    4891 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 14:56:32.472593 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.472574    4891 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 14:56:32.472593 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.472589    4891 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:56:32.472722 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.472686    4891 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 14:56:32.472877 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.472865    4891 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 14:56:32.472908 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.472895    4891 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 14:56:32.472989 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.472980    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 14:56:32.473026 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.472995    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 14:56:32.473026 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.473001    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 14:56:32.473026 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.473006    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 14:56:32.473026 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.473012    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 14:56:32.473026 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.473018    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 14:56:32.473026 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.473023    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 14:56:32.473183 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.473029    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 14:56:32.473183 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.473036    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 14:56:32.473183 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.473043    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 14:56:32.473183 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.473050    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 14:56:32.473183 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.473059    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 14:56:32.473183 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.473083    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 14:56:32.473183 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.473089    4891 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 14:56:32.476373 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.476360    4891 server.go:1263] "Started kubelet"
Nov 02 14:56:32.476721 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.476519    4891 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 14:56:32.476721 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.476512    4891 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 14:56:32.476721 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.476585    4891 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 14:56:32.477001 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.476900    4891 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 14:56:32.476858 ip-10-0-132-27 systemd[1]: Started Kubernetes Kubelet.
Nov 02 14:56:32.477868 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.477847    4891 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 14:56:32.480063 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.480043    4891 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 14:56:32.485417 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.480125    4891 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 14:56:32.485417 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.480214    4891 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:52 +0000 UTC" deadline="2028-06-13 09:25:21.663638214 +0000 UTC"
Nov 02 14:56:32.485417 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.480253    4891 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14130h28m49.183388412s"
Nov 02 14:56:32.485417 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.480307    4891 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 14:56:32.485417 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.480337    4891 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 14:56:32.485417 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.480420    4891 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 14:56:32.485417 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:32.480552    4891 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:56:32.487054 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.486712    4891 factory.go:153] Registering CRI-O factory
Nov 02 14:56:32.487054 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.486735    4891 factory.go:223] Registration of the crio container factory successfully
Nov 02 14:56:32.487054 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.486798    4891 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 14:56:32.487054 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.486816    4891 factory.go:55] Registering systemd factory
Nov 02 14:56:32.487054 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.486824    4891 factory.go:223] Registration of the systemd container factory successfully
Nov 02 14:56:32.487054 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.486840    4891 factory.go:103] Registering Raw factory
Nov 02 14:56:32.487054 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.486855    4891 manager.go:1199] Started watching for new ooms in manager
Nov 02 14:56:32.487951 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.487927    4891 manager.go:319] Starting recovery of all containers
Nov 02 14:56:32.489052 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489023    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" volumeName="kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 14:56:32.489149 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489053    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19ce39d1-e591-4af6-81e4-63d7d7b255c4" volumeName="kubernetes.io/empty-dir/19ce39d1-e591-4af6-81e4-63d7d7b255c4-tmp" seLinuxMountContext=""
Nov 02 14:56:32.489149 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489066    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="885145c5-6077-4435-89ce-73af1e6a5600" volumeName="kubernetes.io/projected/885145c5-6077-4435-89ce-73af1e6a5600-kube-api-access-cshc2" seLinuxMountContext=""
Nov 02 14:56:32.489149 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489080    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acede18e-4d8e-47ee-b31f-918658a9ffba" volumeName="kubernetes.io/configmap/acede18e-4d8e-47ee-b31f-918658a9ffba-serviceca" seLinuxMountContext=""
Nov 02 14:56:32.489149 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489093    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b639a9d2-f451-4cb1-a31f-c3de10b8d3c1" volumeName="kubernetes.io/empty-dir/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-tmp-dir" seLinuxMountContext=""
Nov 02 14:56:32.489149 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489116    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2ebec5-9906-438f-b2ce-010345fa49ba" volumeName="kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovnkube-config" seLinuxMountContext=""
Nov 02 14:56:32.489149 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489128    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2ebec5-9906-438f-b2ce-010345fa49ba" volumeName="kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-env-overrides" seLinuxMountContext=""
Nov 02 14:56:32.489149 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489140    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2ebec5-9906-438f-b2ce-010345fa49ba" volumeName="kubernetes.io/secret/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489154    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="885145c5-6077-4435-89ce-73af1e6a5600" volumeName="kubernetes.io/configmap/885145c5-6077-4435-89ce-73af1e6a5600-telemetry-config" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489166    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bb718bc-429a-4875-bd8d-d52dbba0b252" volumeName="kubernetes.io/projected/9bb718bc-429a-4875-bd8d-d52dbba0b252-kube-api-access-vzqh5" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489179    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7a4115d-9a1a-4ed3-ba75-05c621c1fa63" volumeName="kubernetes.io/projected/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-api-access-s7fb8" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489193    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be0e9509-693a-4b99-b5c5-32770e758b5a" volumeName="kubernetes.io/projected/be0e9509-693a-4b99-b5c5-32770e758b5a-kube-api-access-64mls" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489205    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf04dbe2-4354-4849-959c-33002c7ece68" volumeName="kubernetes.io/configmap/bf04dbe2-4354-4849-959c-33002c7ece68-trusted-ca" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489216    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" volumeName="kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489231    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb94c55-ff22-4e24-99da-047c765a8598" volumeName="kubernetes.io/secret/9fb94c55-ff22-4e24-99da-047c765a8598-cert" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489243    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f25b0184-4053-4694-acc1-c26e9f5602aa" volumeName="kubernetes.io/projected/f25b0184-4053-4694-acc1-c26e9f5602aa-kube-api-access-khdhh" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489255    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" volumeName="kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-kube-api-access-qbskj" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489275    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" volumeName="kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-installation-pull-secrets" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489287    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb39336d-67c0-4cf2-aaff-c873d361f9b2" volumeName="kubernetes.io/configmap/fb39336d-67c0-4cf2-aaff-c873d361f9b2-nginx-conf" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489299    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19ce39d1-e591-4af6-81e4-63d7d7b255c4" volumeName="kubernetes.io/empty-dir/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-tuned" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489351    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74049a08-892a-4ae2-a5a5-161da6182c1c" volumeName="kubernetes.io/projected/74049a08-892a-4ae2-a5a5-161da6182c1c-kube-api-access-96w6k" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489364    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb94c55-ff22-4e24-99da-047c765a8598" volumeName="kubernetes.io/projected/9fb94c55-ff22-4e24-99da-047c765a8598-kube-api-access-xvkkc" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489376    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b639a9d2-f451-4cb1-a31f-c3de10b8d3c1" volumeName="kubernetes.io/configmap/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-config-volume" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489391    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41cff577-8a79-4b9e-89c8-a93e1ecf4ff0" volumeName="kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489403    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063ba34d-7544-410f-b861-5c300cc8f6d4" volumeName="kubernetes.io/configmap/063ba34d-7544-410f-b861-5c300cc8f6d4-service-ca-bundle" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489415    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063ba34d-7544-410f-b861-5c300cc8f6d4" volumeName="kubernetes.io/empty-dir/063ba34d-7544-410f-b861-5c300cc8f6d4-tmp" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489426    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063ba34d-7544-410f-b861-5c300cc8f6d4" volumeName="kubernetes.io/projected/063ba34d-7544-410f-b861-5c300cc8f6d4-kube-api-access-ftvbk" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489438    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="234a4a99-5491-44ef-b58c-3e4ab32584e5" volumeName="kubernetes.io/secret/234a4a99-5491-44ef-b58c-3e4ab32584e5-serving-cert" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489455    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ff63dfc-17d4-4efc-a23a-d2e46a6885fc" volumeName="kubernetes.io/configmap/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-config" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489468    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d2ae30e-0191-41ff-85da-dedf17ad27b2" volumeName="kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-certificates" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489494    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d2ae30e-0191-41ff-85da-dedf17ad27b2" volumeName="kubernetes.io/empty-dir/8d2ae30e-0191-41ff-85da-dedf17ad27b2-ca-trust-extracted" seLinuxMountContext=""
Nov 02 14:56:32.489500 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489507    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" volumeName="kubernetes.io/projected/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-kube-api-access-b8n6k" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489519    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2ebec5-9906-438f-b2ce-010345fa49ba" volumeName="kubernetes.io/projected/6f2ebec5-9906-438f-b2ce-010345fa49ba-kube-api-access-rmfrd" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489531    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eaf7fe9-c300-4798-8212-e57e5a9cb03d" volumeName="kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489544    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf04dbe2-4354-4849-959c-33002c7ece68" volumeName="kubernetes.io/configmap/bf04dbe2-4354-4849-959c-33002c7ece68-config" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489556    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf04dbe2-4354-4849-959c-33002c7ece68" volumeName="kubernetes.io/projected/bf04dbe2-4354-4849-959c-33002c7ece68-kube-api-access-9l647" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489568    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" volumeName="kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-bound-sa-token" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489579    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0290bc25-3e34-4d43-9215-f15cfc44f686" volumeName="kubernetes.io/projected/0290bc25-3e34-4d43-9215-f15cfc44f686-kube-api-access-flmc9" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489591    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" volumeName="kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-certificates" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489815    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" volumeName="kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-trusted-ca" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489835    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bcba632-1434-4533-951f-84b8bba1f0d8" volumeName="kubernetes.io/projected/1bcba632-1434-4533-951f-84b8bba1f0d8-kube-api-access-psj4b" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489848    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74049a08-892a-4ae2-a5a5-161da6182c1c" volumeName="kubernetes.io/empty-dir/74049a08-892a-4ae2-a5a5-161da6182c1c-tmp-dir" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489859    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf04dbe2-4354-4849-959c-33002c7ece68" volumeName="kubernetes.io/secret/bf04dbe2-4354-4849-959c-33002c7ece68-serving-cert" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489871    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63faa28d-8806-4332-99c4-9f715a0b1a86" volumeName="kubernetes.io/projected/63faa28d-8806-4332-99c4-9f715a0b1a86-kube-api-access-kg79x" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489882    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063ba34d-7544-410f-b861-5c300cc8f6d4" volumeName="kubernetes.io/empty-dir/063ba34d-7544-410f-b861-5c300cc8f6d4-snapshots" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489896    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" volumeName="kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cni-binary-copy" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489907    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="100e3924-bcfc-4208-84ea-c1ff1656e949" volumeName="kubernetes.io/configmap/100e3924-bcfc-4208-84ea-c1ff1656e949-cni-binary-copy" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489920    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bea213e-5ff8-447d-98db-fa0ec51cda10" volumeName="kubernetes.io/projected/9bea213e-5ff8-447d-98db-fa0ec51cda10-kube-api-access-bx97c" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489931    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7d93f72-6fc0-469a-8c16-9feed09bda86" volumeName="kubernetes.io/configmap/c7d93f72-6fc0-469a-8c16-9feed09bda86-konnectivity-ca" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489942    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" volumeName="kubernetes.io/empty-dir/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-ca-trust-extracted" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489953    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc12a767-3259-4dc0-a809-d123b58334e3" volumeName="kubernetes.io/configmap/fc12a767-3259-4dc0-a809-d123b58334e3-config" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489966    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19ce39d1-e591-4af6-81e4-63d7d7b255c4" volumeName="kubernetes.io/projected/19ce39d1-e591-4af6-81e4-63d7d7b255c4-kube-api-access-dsk26" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.489977    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bcba632-1434-4533-951f-84b8bba1f0d8" volumeName="kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-stats-auth" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490001    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="234a4a99-5491-44ef-b58c-3e4ab32584e5" volumeName="kubernetes.io/configmap/234a4a99-5491-44ef-b58c-3e4ab32584e5-config" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490012    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="234a4a99-5491-44ef-b58c-3e4ab32584e5" volumeName="kubernetes.io/projected/234a4a99-5491-44ef-b58c-3e4ab32584e5-kube-api-access-2znqw" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490024    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2ebec5-9906-438f-b2ce-010345fa49ba" volumeName="kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490045    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d2ae30e-0191-41ff-85da-dedf17ad27b2" volumeName="kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-bound-sa-token" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490058    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b639a9d2-f451-4cb1-a31f-c3de10b8d3c1" volumeName="kubernetes.io/projected/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-kube-api-access-dtqv7" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490070    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc12a767-3259-4dc0-a809-d123b58334e3" volumeName="kubernetes.io/projected/fc12a767-3259-4dc0-a809-d123b58334e3-kube-api-access-klllg" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490080    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="100e3924-bcfc-4208-84ea-c1ff1656e949" volumeName="kubernetes.io/configmap/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-daemon-config" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490094    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27c6485e-f8bc-4b94-839c-746297629798" volumeName="kubernetes.io/projected/27c6485e-f8bc-4b94-839c-746297629798-kube-api-access-87hwb" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490106    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d2ae30e-0191-41ff-85da-dedf17ad27b2" volumeName="kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490122    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d2ae30e-0191-41ff-85da-dedf17ad27b2" volumeName="kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-installation-pull-secrets" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490141    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7a4115d-9a1a-4ed3-ba75-05c621c1fa63" volumeName="kubernetes.io/empty-dir/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-data-volume" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490154    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7d93f72-6fc0-469a-8c16-9feed09bda86" volumeName="kubernetes.io/secret/c7d93f72-6fc0-469a-8c16-9feed09bda86-agent-certs" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490170    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="100e3924-bcfc-4208-84ea-c1ff1656e949" volumeName="kubernetes.io/projected/100e3924-bcfc-4208-84ea-c1ff1656e949-kube-api-access-npd9z" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490182    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063ba34d-7544-410f-b861-5c300cc8f6d4" volumeName="kubernetes.io/configmap/063ba34d-7544-410f-b861-5c300cc8f6d4-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490194    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="063ba34d-7544-410f-b861-5c300cc8f6d4" volumeName="kubernetes.io/secret/063ba34d-7544-410f-b861-5c300cc8f6d4-serving-cert" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490207    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ff63dfc-17d4-4efc-a23a-d2e46a6885fc" volumeName="kubernetes.io/projected/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-kube-api-access-z9tf9" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490219    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acede18e-4d8e-47ee-b31f-918658a9ffba" volumeName="kubernetes.io/projected/acede18e-4d8e-47ee-b31f-918658a9ffba-kube-api-access-7n9ql" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490233    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bcba632-1434-4533-951f-84b8bba1f0d8" volumeName="kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-default-certificate" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490247    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49e40d7f-ab6a-4817-ba3e-3e2ab361b454" volumeName="kubernetes.io/projected/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-kube-api-access-k2pbg" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490262    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d2ae30e-0191-41ff-85da-dedf17ad27b2" volumeName="kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-trusted-ca" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490276    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d2ae30e-0191-41ff-85da-dedf17ad27b2" volumeName="kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-kube-api-access-2fwhf" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490291    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bb718bc-429a-4875-bd8d-d52dbba0b252" volumeName="kubernetes.io/configmap/9bb718bc-429a-4875-bd8d-d52dbba0b252-iptables-alerter-script" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490305    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" volumeName="kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490336    4891 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc12a767-3259-4dc0-a809-d123b58334e3" volumeName="kubernetes.io/secret/fc12a767-3259-4dc0-a809-d123b58334e3-serving-cert" seLinuxMountContext=""
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490350    4891 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 14:56:32.490723 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.490357    4891 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 14:56:32.500085 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.500063    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/1.log"
Nov 02 14:56:32.500250 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.500093    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/1.log"
Nov 02 14:56:32.511609 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.511578    4891 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 14:56:32.513908 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.513883    4891 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 14:56:32.514001 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.513912    4891 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 14:56:32.514001 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.513936    4891 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 14:56:32.514001 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:32.513977    4891 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 14:56:32.520850 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.520825    4891 generic.go:356] "Generic (PLEG): container finished" podID="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" containerID="ef6d4ace3327cea9913c8bdf3e7b860c4d9bfb43128480d3666ccc19c3d04016" exitCode=0
Nov 02 14:56:32.520850 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.520848    4891 generic.go:356] "Generic (PLEG): container finished" podID="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" containerID="a988f4d8c1a9eb03c33c6d85dc02da88d5616e0cc5a0285e669bfa641a591559" exitCode=0
Nov 02 14:56:32.520986 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.520857    4891 generic.go:356] "Generic (PLEG): container finished" podID="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" containerID="f13140e6b9cc2cf72e46f17da6f3d3cdd6390c4b36a8f49e2e9af8c0c358557b" exitCode=0
Nov 02 14:56:32.520986 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.520865    4891 generic.go:356] "Generic (PLEG): container finished" podID="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" containerID="b2474e295bfaee035affd5a28a8afda8ad13c822e2d5df4f32fb7b8bd61cf5e9" exitCode=0
Nov 02 14:56:32.520986 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.520872    4891 generic.go:356] "Generic (PLEG): container finished" podID="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" containerID="2d81250c08e43e7bd99007969cf73fee39a02a82370c16ed50ec5b2d00010977" exitCode=0
Nov 02 14:56:32.520986 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.520879    4891 generic.go:356] "Generic (PLEG): container finished" podID="2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac" containerID="93ab6e9ef87f647231a4772af058c277884e6d6e860f27bb25cb9937738a7d94" exitCode=0
Nov 02 14:56:32.524330 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.524289    4891 generic.go:356] "Generic (PLEG): container finished" podID="632223b57cdaa2366b6d306d2b0b8330" containerID="e65cb548c4a51c2f73fae4d1da4ab7dbfe8a734339eac18b2a77e497bfc4f1c2" exitCode=0
Nov 02 14:56:32.553945 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.553921    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/1.log"
Nov 02 14:56:32.554054 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.553951    4891 generic.go:356] "Generic (PLEG): container finished" podID="bf04dbe2-4354-4849-959c-33002c7ece68" containerID="e65d58d2a1b20beec085715ef00a07f1555863222f9e3afe972c3aa920641f19" exitCode=255
Nov 02 14:56:32.570637 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.570617    4891 manager.go:324] Recovery completed
Nov 02 14:56:32.581426 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:32.581403    4891 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:56:32.587000 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.586983    4891 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:56:32.588625 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.588601    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:56:32.588716 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.588639    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:56:32.588716 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.588662    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:56:32.591658 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.591633    4891 cpu_manager.go:226] "Starting" policy="none"
Nov 02 14:56:32.591658 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.591656    4891 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 14:56:32.591769 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.591663    4891 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:56:32.591769 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.591674    4891 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 14:56:32.591905 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.591807    4891 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 14:56:32.591905 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.591818    4891 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 14:56:32.591905 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.591840    4891 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 14:56:32.591905 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.591848    4891 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 14:56:32.591905 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.591855    4891 policy_none.go:50] "Start"
Nov 02 14:56:32.591905 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.591867    4891 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 14:56:32.591905 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.591880    4891 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 14:56:32.592117 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.592017    4891 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 14:56:32.592117 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.592024    4891 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 14:56:32.592117 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.592030    4891 policy_none.go:44] "Start"
Nov 02 14:56:32.592665 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.592650    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:56:32.592752 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.592681    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:56:32.592752 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.592697    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:56:32.596365 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.596343    4891 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 14:56:32.598995 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.598982    4891 manager.go:341] "Starting Device Plugin manager"
Nov 02 14:56:32.599054 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:32.599007    4891 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 14:56:32.599054 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.599027    4891 server.go:87] "Starting device plugin registration server"
Nov 02 14:56:32.599280 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.599269    4891 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 14:56:32.599351 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.599280    4891 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 14:56:32.599400 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.599374    4891 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 14:56:32.599503 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.599478    4891 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 14:56:32.599565 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.599509    4891 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 14:56:32.599565 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.599516    4891 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 14:56:32.599664 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.599569    4891 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:56:32.599500783Z","UUID":"df1afaa5-a628-43b0-8018-b0b1aae44c86","Handler":null,"Name":"","Endpoint":""}
Nov 02 14:56:32.601014 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:32.600995    4891 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 14:56:32.601118 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:32.601028    4891 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-27.us-west-1.compute.internal\" not found"
Nov 02 14:56:32.601180 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.601124    4891 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 14:56:32.601180 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.601151    4891 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 14:56:32.614706 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.614263    4891 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"]
Nov 02 14:56:32.614798 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.614788    4891 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:56:32.615806 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.615789    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:56:32.615886 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.615820    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:56:32.615886 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.615834    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:56:32.616159 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.616145    4891 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:56:32.616280 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.616266    4891 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:56:32.617039 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.617020    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:56:32.617112 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.617048    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:56:32.617112 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.617058    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:56:32.617112 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.617025    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:56:32.617242 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.617123    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:56:32.617242 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.617139    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:56:32.617242 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:32.617187    4891 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found" node="ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.617491 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.617446    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal" event={"ID":"632223b57cdaa2366b6d306d2b0b8330","Type":"ContainerStarted","Data":"b50edc609036e72144ffeedc1ae7c935ba74ff9472774fa2a6571dd38b9c0bcc"}
Nov 02 14:56:32.617536 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.617505    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal" event={"ID":"632223b57cdaa2366b6d306d2b0b8330","Type":"ContainerDied","Data":"e65cb548c4a51c2f73fae4d1da4ab7dbfe8a734339eac18b2a77e497bfc4f1c2"}
Nov 02 14:56:32.617536 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.617527    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal" event={"ID":"632223b57cdaa2366b6d306d2b0b8330","Type":"ContainerStarted","Data":"365b3df95f728f6953f143442eb4f2a78449283c2a46402d6fac2c87491832dc"}
Nov 02 14:56:32.617590 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.617559    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal" event={"ID":"c52ee3e32e488dfc59e1227cdb8da56d","Type":"ContainerStarted","Data":"c9850c15758c890712e163c210ca46b74a0f8c176c542b704a4c552b62c6d05d"}
Nov 02 14:56:32.617590 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.617563    4891 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:56:32.617590 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.617577    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal" event={"ID":"c52ee3e32e488dfc59e1227cdb8da56d","Type":"ContainerStarted","Data":"9a07090f931f3077f7a532bbdd8e28e8c071f4f7870bcecd391fabb7ea9cdbc0"}
Nov 02 14:56:32.618445 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.618431    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:56:32.618497 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.618458    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:56:32.618497 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.618471    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:56:32.618603 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:32.618589    4891 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-27.us-west-1.compute.internal\" not found" node="ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.676200 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.676172    4891 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 14:56:32.678734 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.678717    4891 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:56:32.680450 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.680433    4891 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:56:32.680638 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.680624    4891 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:56:32.681616 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.681604    4891 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.699773 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.699748    4891 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:56:32.701860 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.701842    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 14:56:32.701926 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.701872    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 14:56:32.701926 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.701884    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 14:56:32.701926 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.701926    4891 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.731363 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.731098    4891 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:56:32.770809 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.770783    4891 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:56:32.770938 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:32.770840    4891 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.770938 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.770854    4891 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.776726 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.776705    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/632223b57cdaa2366b6d306d2b0b8330-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"632223b57cdaa2366b6d306d2b0b8330\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.776775 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.776734    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c52ee3e32e488dfc59e1227cdb8da56d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"c52ee3e32e488dfc59e1227cdb8da56d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.776775 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.776753    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/632223b57cdaa2366b6d306d2b0b8330-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"632223b57cdaa2366b6d306d2b0b8330\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.834153 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.834129    4891 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.834273 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.834200    4891 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.834793 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.834777    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeNotReady"
Nov 02 14:56:32.834859 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.834809    4891 setters.go:546] "Node became not ready" node="ip-10-0-132-27.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T14:56:32Z","lastTransitionTime":"2026-11-02T14:56:32Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 02 14:56:32.842562 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.842543    4891 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:56:32.842636 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:32.842590    4891 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.847598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.847582    4891 nodeinfomanager.go:471] existing CSINode "ip-10-0-132-27.us-west-1.compute.internal" is owned by different node (oldNodeID="923460f4-c10b-4d62-9c26-51a04c3008ef", newNodeID=""), cleaning up...
Nov 02 14:56:32.877804 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.877778    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/632223b57cdaa2366b6d306d2b0b8330-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"632223b57cdaa2366b6d306d2b0b8330\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.877902 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.877732    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/632223b57cdaa2366b6d306d2b0b8330-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"632223b57cdaa2366b6d306d2b0b8330\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.877902 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.877838    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/632223b57cdaa2366b6d306d2b0b8330-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"632223b57cdaa2366b6d306d2b0b8330\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.877902 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.877858    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c52ee3e32e488dfc59e1227cdb8da56d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"c52ee3e32e488dfc59e1227cdb8da56d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.877993 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.877906    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c52ee3e32e488dfc59e1227cdb8da56d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"c52ee3e32e488dfc59e1227cdb8da56d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:32.877993 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:32.877908    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/632223b57cdaa2366b6d306d2b0b8330-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" (UID: \"632223b57cdaa2366b6d306d2b0b8330\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:33.467733 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.467702    4891 apiserver.go:52] "Watching apiserver"
Nov 02 14:56:33.564453 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.564427    4891 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:33.564596 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.564551    4891 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:33.620829 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.620796    4891 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 14:56:33.621484 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.621459    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rtsrv","openshift-cluster-node-tuning-operator/tuned-5lhfv","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-nx6px","openshift-ingress-canary/ingress-canary-jcqwk","openshift-dns/node-resolver-cllms","openshift-multus/multus-w5w2m","openshift-network-operator/iptables-alerter-whl44","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l","openshift-dns/dns-default-f6fgv","kube-system/kas-connection-checker-685895c7c9-xx5tn","openshift-insights/insights-runtime-extractor-dtxzd","kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t","openshift-network-diagnostics/network-check-target-jjhkj","openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x","openshift-multus/network-metrics-daemon-rkxt7","openshift-image-registry/image-registry-dcff7b7b8-bck5q","kube-system/kas-connection-checker-685895c7c9-v4f7p","openshift-ingress/router-default-684675bd5f-zwwbp","openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb","openshift-image-registry/image-registry-74fd4fbb69-thwc5","kube-system/kas-connection-checker-685895c7c9-64bm4","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m","openshift-image-registry/node-ca-9htnf","openshift-ovn-kubernetes/ovnkube-node-8822k","openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q","openshift-insights/insights-operator-7c4986d449-ggtvw","openshift-network-console/networking-console-plugin-86c478bb88-mf5tv","kube-system/konnectivity-agent-spbjm","openshift-multus/multus-additional-cni-plugins-596b4"]
Nov 02 14:56:33.623213 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.623188    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:33.623639 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.623620    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:33.624354 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.624336    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:33.624623 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.624600    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.626090 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.626070    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:33.626173 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.626118    4891 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:56:33.626173 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.626164    4891 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:33.627126 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.627105    4891 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:56:33.627199 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.627162    4891 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-27.us-west-1.compute.internal"
Nov 02 14:56:33.630027 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.630006    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:33.634484 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.634465    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.635383 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.635366    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:33.635906 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.635887    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:33.674840 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.674818    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.674936 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.674868    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.675415 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.675396    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.675485 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.675452    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 14:56:33.675546 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.675533    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.675890 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.675873    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 14:56:33.675890 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.675878    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.677064 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.677043    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 14:56:33.677064 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.677046    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 14:56:33.677574 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.677527    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 14:56:33.677644 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.677622    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 14:56:33.677682 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.677641    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 14:56:33.677682 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.677664    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 14:56:33.677770 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.677758    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-p249k\""
Nov 02 14:56:33.679378 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679362    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rtl8l\""
Nov 02 14:56:33.679637 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679615    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-b2j82\""
Nov 02 14:56:33.679747 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679690    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.679812 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679746    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 14:56:33.679812 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679754    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 14:56:33.679812 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679795    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 14:56:33.679812 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679807    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4ks69\""
Nov 02 14:56:33.679967 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679831    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.679967 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679868    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.679967 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679868    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 14:56:33.679967 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679885    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.679967 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679887    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 14:56:33.679967 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679940    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.680157 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.679962    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.680157 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.680046    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.680665 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.680651    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 14:56:33.681691 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.681667    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/063ba34d-7544-410f-b861-5c300cc8f6d4-snapshots\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:33.681735 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.681697    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-cni-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.681775 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.681729    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/100e3924-bcfc-4208-84ea-c1ff1656e949-cni-binary-copy\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.681859 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.681841    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/063ba34d-7544-410f-b861-5c300cc8f6d4-snapshots\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:33.681908 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.681894    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.682031 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682015    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-run\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.682082 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682054    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9ql\" (UniqueName: \"kubernetes.io/projected/acede18e-4d8e-47ee-b31f-918658a9ffba-kube-api-access-7n9ql\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:56:33.682130 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682083    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-ca-trust-extracted\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.682130 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682109    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-socket-dir-parent\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.682233 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682191    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-ca-trust-extracted\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.682282 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682258    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/063ba34d-7544-410f-b861-5c300cc8f6d4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:33.682340 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682284    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysctl-d\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.682340 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682303    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-tuned\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.682437 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682374    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-tuned\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.682437 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682385    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74049a08-892a-4ae2-a5a5-161da6182c1c-hosts-file\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:56:33.682437 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682416    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96w6k\" (UniqueName: \"kubernetes.io/projected/74049a08-892a-4ae2-a5a5-161da6182c1c-kube-api-access-96w6k\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:56:33.682539 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682443    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acede18e-4d8e-47ee-b31f-918658a9ffba-host\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:56:33.682539 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682481    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc12a767-3259-4dc0-a809-d123b58334e3-config\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:33.682667 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682652    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:33.682728 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682687    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-var-lib-kubelet\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.682728 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682713    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-trusted-ca\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.682800 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682771    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc12a767-3259-4dc0-a809-d123b58334e3-serving-cert\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:33.682833 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682814    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fc12a767-3259-4dc0-a809-d123b58334e3-config\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:33.682863 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682838    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9tf9\" (UniqueName: \"kubernetes.io/projected/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-kube-api-access-z9tf9\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:33.682863 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682858    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87hwb\" (UniqueName: \"kubernetes.io/projected/27c6485e-f8bc-4b94-839c-746297629798-kube-api-access-87hwb\") pod \"volume-data-source-validator-58c9f5f46d-96v9l\" (UID: \"27c6485e-f8bc-4b94-839c-746297629798\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l"
Nov 02 14:56:33.682930 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682912    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/063ba34d-7544-410f-b861-5c300cc8f6d4-serving-cert\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:33.682997 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682978    4891 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 14:56:33.683046 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683011    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:33.683046 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682985    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-config-volume\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:33.683124 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.682983    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc12a767-3259-4dc0-a809-d123b58334e3-serving-cert\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:33.683124 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683059    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-tmp-dir\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:33.683124 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683084    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtqv7\" (UniqueName: \"kubernetes.io/projected/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-kube-api-access-dtqv7\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:33.683124 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683107    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/234a4a99-5491-44ef-b58c-3e4ab32584e5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:33.683124 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683122    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d2ae30e-0191-41ff-85da-dedf17ad27b2-ca-trust-extracted\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.683343 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683141    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.683343 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683148    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-tmp-dir\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:33.683343 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683159    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:33.683343 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683195    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf04dbe2-4354-4849-959c-33002c7ece68-trusted-ca\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:33.683343 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683213    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-installation-pull-secrets\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.683343 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683232    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d2ae30e-0191-41ff-85da-dedf17ad27b2-ca-trust-extracted\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.683343 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683250    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-kubernetes\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.683343 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683258    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/234a4a99-5491-44ef-b58c-3e4ab32584e5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:33.683343 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683280    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063ba34d-7544-410f-b861-5c300cc8f6d4-tmp\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:33.683343 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683340    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bb718bc-429a-4875-bd8d-d52dbba0b252-host-slash\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:56:33.683711 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683365    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzqh5\" (UniqueName: \"kubernetes.io/projected/9bb718bc-429a-4875-bd8d-d52dbba0b252-kube-api-access-vzqh5\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:56:33.683711 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683374    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063ba34d-7544-410f-b861-5c300cc8f6d4-tmp\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:33.683711 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683411    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysconfig\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.683711 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683482    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf04dbe2-4354-4849-959c-33002c7ece68-trusted-ca\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:33.683711 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683534    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-systemd\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.683711 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683560    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-lib-modules\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.683711 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683584    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-host\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.683711 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683613    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysctl-conf\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.683711 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683634    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ce39d1-e591-4af6-81e4-63d7d7b255c4-tmp\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.683711 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683675    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74049a08-892a-4ae2-a5a5-161da6182c1c-tmp-dir\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:56:33.683711 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683700    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cni-binary-copy\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.683711 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683717    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683736    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bb718bc-429a-4875-bd8d-d52dbba0b252-iptables-alerter-script\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683745    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ce39d1-e591-4af6-81e4-63d7d7b255c4-tmp\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683752    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klllg\" (UniqueName: \"kubernetes.io/projected/fc12a767-3259-4dc0-a809-d123b58334e3-kube-api-access-klllg\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683767    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74049a08-892a-4ae2-a5a5-161da6182c1c-tmp-dir\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683777    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-certificates\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683819    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683842    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-modprobe-d\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683866    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-image-registry-private-configuration\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683896    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acede18e-4d8e-47ee-b31f-918658a9ffba-serviceca\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683973    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-bound-sa-token\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.683994    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l647\" (UniqueName: \"kubernetes.io/projected/bf04dbe2-4354-4849-959c-33002c7ece68-kube-api-access-9l647\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684009    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb39336d-67c0-4cf2-aaff-c873d361f9b2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684033    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-k8s-cni-cncf-io\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684069    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-system-cni-dir\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684105    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/063ba34d-7544-410f-b861-5c300cc8f6d4-service-ca-bundle\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684159    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-sys\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684177    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-bound-sa-token\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684191    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf04dbe2-4354-4849-959c-33002c7ece68-config\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684211    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf04dbe2-4354-4849-959c-33002c7ece68-serving-cert\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:33.684215 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684229    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-image-registry-private-configuration\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684245    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-trusted-ca\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684264    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8n6k\" (UniqueName: \"kubernetes.io/projected/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-kube-api-access-b8n6k\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684289    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx97c\" (UniqueName: \"kubernetes.io/projected/9bea213e-5ff8-447d-98db-fa0ec51cda10-kube-api-access-bx97c\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684335    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/234a4a99-5491-44ef-b58c-3e4ab32584e5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684366    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2znqw\" (UniqueName: \"kubernetes.io/projected/234a4a99-5491-44ef-b58c-3e4ab32584e5-kube-api-access-2znqw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684402    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf04dbe2-4354-4849-959c-33002c7ece68-serving-cert\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684460    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf04dbe2-4354-4849-959c-33002c7ece68-config\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684480    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684520    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/234a4a99-5491-44ef-b58c-3e4ab32584e5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684521    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbskj\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-kube-api-access-qbskj\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684565    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-system-cni-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684581    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-netns\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684642    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftvbk\" (UniqueName: \"kubernetes.io/projected/063ba34d-7544-410f-b861-5c300cc8f6d4-kube-api-access-ftvbk\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684677    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsk26\" (UniqueName: \"kubernetes.io/projected/19ce39d1-e591-4af6-81e4-63d7d7b255c4-kube-api-access-dsk26\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684703    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw78t\" (UniqueName: \"kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t\") pod \"network-check-target-jjhkj\" (UID: \"8eaf7fe9-c300-4798-8212-e57e5a9cb03d\") " pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684727    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mls\" (UniqueName: \"kubernetes.io/projected/be0e9509-693a-4b99-b5c5-32770e758b5a-kube-api-access-64mls\") pod \"network-check-source-6f78d59f99-cnq6q\" (UID: \"be0e9509-693a-4b99-b5c5-32770e758b5a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684753    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdhh\" (UniqueName: \"kubernetes.io/projected/f25b0184-4053-4694-acc1-c26e9f5602aa-kube-api-access-khdhh\") pod \"kas-connection-checker-685895c7c9-v4f7p\" (UID: \"f25b0184-4053-4694-acc1-c26e9f5602aa\") " pod="kube-system/kas-connection-checker-685895c7c9-v4f7p"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684782    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cnibin\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684807    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684833    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/885145c5-6077-4435-89ce-73af1e6a5600-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:33.684911 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684865    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshc2\" (UniqueName: \"kubernetes.io/projected/885145c5-6077-4435-89ce-73af1e6a5600-kube-api-access-cshc2\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:33.685499 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684925    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:33.685499 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.684999    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.685499 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.685021    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-certificates\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.685499 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.685075    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-installation-pull-secrets\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.685499 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.685108    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwhf\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-kube-api-access-2fwhf\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.685499 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.685127    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:33.685499 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.685149    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-cnibin\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.720831 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.720772    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 14:56:33.721978 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.721940    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5rstk\""
Nov 02 14:56:33.722811 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.722796    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 14:56:33.723189 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.723176    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 14:56:33.723361 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.723351    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.723892 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.723880    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.723979 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.723960    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-certificates\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.724147 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.724131    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-image-registry-private-configuration\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.724244 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.724226    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acede18e-4d8e-47ee-b31f-918658a9ffba-serviceca\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:56:33.724477 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.724462    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-image-registry-private-configuration\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.725193 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.725176    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-certificates\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.725379 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.725364    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.725425 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.725374    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 14:56:33.725425 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.725422    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.725480 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.725465    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5lvgl\""
Nov 02 14:56:33.730256 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.730231    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8dn6m\""
Nov 02 14:56:33.730359 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.730239    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 14:56:33.730359 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.730329    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 14:56:33.730546 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.730527    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mnrgm\""
Nov 02 14:56:33.730778 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.730760    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.730838 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.730784    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 14:56:33.730890 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.730850    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 14:56:33.730890 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.730760    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 14:56:33.730996 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.730978    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.732998 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.732971    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 14:56:33.733094 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.733079    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.733225 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.733210    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-snsjj\""
Nov 02 14:56:33.733346 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.733333    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.733425 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.733414    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.733522 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.733510    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.733678 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.733664    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 14:56:33.733760 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.733749    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.733978 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.733957    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.734136 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734113    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.734223 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734205    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.734280 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734247    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.734362 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734340    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 14:56:33.734419 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734379    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.734473 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734417    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 14:56:33.734522 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734493    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.734522 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734519    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 14:56:33.734617 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734541    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 14:56:33.734617 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734597    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rrdxh\""
Nov 02 14:56:33.734617 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734605    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.734752 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734614    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/063ba34d-7544-410f-b861-5c300cc8f6d4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:33.734752 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734631    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 14:56:33.734752 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734615    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-trusted-ca\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.734752 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734664    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m4lkk\""
Nov 02 14:56:33.734752 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734674    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 14:56:33.734752 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.733958    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 14:56:33.734752 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734670    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 14:56:33.734752 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734734    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.734752 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.734745    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.735497 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.735267    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 14:56:33.735497 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.735304    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-trusted-ca\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.735497 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.735345    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-installation-pull-secrets\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.735497 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.735369    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-installation-pull-secrets\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.735497 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.735382    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/885145c5-6077-4435-89ce-73af1e6a5600-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:33.735497 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.735402    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.735497 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.735420    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 14:56:33.735497 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.735458    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/063ba34d-7544-410f-b861-5c300cc8f6d4-service-ca-bundle\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:33.735497 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.735492    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.735924 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.735494    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-44bxs\""
Nov 02 14:56:33.735924 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.735632    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/100e3924-bcfc-4208-84ea-c1ff1656e949-cni-binary-copy\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.743128 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.743113    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/063ba34d-7544-410f-b861-5c300cc8f6d4-serving-cert\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:33.743255 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.743238    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-config-volume\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:33.744347 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.744294    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb39336d-67c0-4cf2-aaff-c873d361f9b2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:33.744347 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.744322    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cni-binary-copy\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.744451 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.744399    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bb718bc-429a-4875-bd8d-d52dbba0b252-iptables-alerter-script\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:56:33.770442 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.770423    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 14:56:33.770442 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.770439    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.770568 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.770427    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 14:56:33.770568 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.770474    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 14:56:33.770568 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.770487    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 14:56:33.772556 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.772540    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 14:56:33.772623 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.772566    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 14:56:33.772623 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.772580    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 14:56:33.772623 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.772590    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-jb9s2\""
Nov 02 14:56:33.772902 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.772878    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dhf26\""
Nov 02 14:56:33.773016 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773000    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 14:56:33.773080 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773011    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 14:56:33.773080 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773015    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9nw68\""
Nov 02 14:56:33.773080 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773029    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ptvxg\""
Nov 02 14:56:33.773080 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773031    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bm8rt\""
Nov 02 14:56:33.773080 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773061    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fbnz4\""
Nov 02 14:56:33.773353 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773287    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zct6j\""
Nov 02 14:56:33.773353 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773294    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5w9z9\""
Nov 02 14:56:33.773353 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773341    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nw765\""
Nov 02 14:56:33.773353 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773355    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wtdgk\""
Nov 02 14:56:33.773353 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773357    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dh847\""
Nov 02 14:56:33.773353 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773357    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 14:56:33.773353 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773388    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zp9vx\""
Nov 02 14:56:33.773716 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773418    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.773716 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773472    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 14:56:33.774035 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773953    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8n6k\" (UniqueName: \"kubernetes.io/projected/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-kube-api-access-b8n6k\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.774035 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773964    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzqh5\" (UniqueName: \"kubernetes.io/projected/9bb718bc-429a-4875-bd8d-d52dbba0b252-kube-api-access-vzqh5\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:56:33.774035 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.773989    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96w6k\" (UniqueName: \"kubernetes.io/projected/74049a08-892a-4ae2-a5a5-161da6182c1c-kube-api-access-96w6k\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:56:33.774254 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.774082    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtqv7\" (UniqueName: \"kubernetes.io/projected/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-kube-api-access-dtqv7\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:33.774338 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.774300    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwhf\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-kube-api-access-2fwhf\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.774611 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.774591    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9ql\" (UniqueName: \"kubernetes.io/projected/acede18e-4d8e-47ee-b31f-918658a9ffba-kube-api-access-7n9ql\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:56:33.774743 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.774700    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-bound-sa-token\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.774850 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.774826    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klllg\" (UniqueName: \"kubernetes.io/projected/fc12a767-3259-4dc0-a809-d123b58334e3-kube-api-access-klllg\") pod \"service-ca-operator-544d749c8c-wbwdb\" (UID: \"fc12a767-3259-4dc0-a809-d123b58334e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb"
Nov 02 14:56:33.774945 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.774925    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw78t\" (UniqueName: \"kubernetes.io/projected/8eaf7fe9-c300-4798-8212-e57e5a9cb03d-kube-api-access-hw78t\") pod \"network-check-target-jjhkj\" (UID: \"8eaf7fe9-c300-4798-8212-e57e5a9cb03d\") " pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:33.775022 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.775006    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9tf9\" (UniqueName: \"kubernetes.io/projected/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-kube-api-access-z9tf9\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:33.775102 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.775019    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87hwb\" (UniqueName: \"kubernetes.io/projected/27c6485e-f8bc-4b94-839c-746297629798-kube-api-access-87hwb\") pod \"volume-data-source-validator-58c9f5f46d-96v9l\" (UID: \"27c6485e-f8bc-4b94-839c-746297629798\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-96v9l"
Nov 02 14:56:33.775102 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.775055    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftvbk\" (UniqueName: \"kubernetes.io/projected/063ba34d-7544-410f-b861-5c300cc8f6d4-kube-api-access-ftvbk\") pod \"insights-operator-7c4986d449-ggtvw\" (UID: \"063ba34d-7544-410f-b861-5c300cc8f6d4\") " pod="openshift-insights/insights-operator-7c4986d449-ggtvw"
Nov 02 14:56:33.775234 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.775211    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdhh\" (UniqueName: \"kubernetes.io/projected/f25b0184-4053-4694-acc1-c26e9f5602aa-kube-api-access-khdhh\") pod \"kas-connection-checker-685895c7c9-v4f7p\" (UID: \"f25b0184-4053-4694-acc1-c26e9f5602aa\") " pod="kube-system/kas-connection-checker-685895c7c9-v4f7p"
Nov 02 14:56:33.775382 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.775363    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshc2\" (UniqueName: \"kubernetes.io/projected/885145c5-6077-4435-89ce-73af1e6a5600-kube-api-access-cshc2\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:33.775543 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.775523    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2znqw\" (UniqueName: \"kubernetes.io/projected/234a4a99-5491-44ef-b58c-3e4ab32584e5-kube-api-access-2znqw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lz25d\" (UID: \"234a4a99-5491-44ef-b58c-3e4ab32584e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d"
Nov 02 14:56:33.775625 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.775605    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mls\" (UniqueName: \"kubernetes.io/projected/be0e9509-693a-4b99-b5c5-32770e758b5a-kube-api-access-64mls\") pod \"network-check-source-6f78d59f99-cnq6q\" (UID: \"be0e9509-693a-4b99-b5c5-32770e758b5a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cnq6q"
Nov 02 14:56:33.775673 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.775639    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-bound-sa-token\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.775705 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.775684    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsk26\" (UniqueName: \"kubernetes.io/projected/19ce39d1-e591-4af6-81e4-63d7d7b255c4-kube-api-access-dsk26\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.775741 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.775711    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbskj\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-kube-api-access-qbskj\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.775825 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.775812    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l647\" (UniqueName: \"kubernetes.io/projected/bf04dbe2-4354-4849-959c-33002c7ece68-kube-api-access-9l647\") pod \"console-operator-7cd56b8dbc-rtsrv\" (UID: \"bf04dbe2-4354-4849-959c-33002c7ece68\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:33.776579 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.776562    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx97c\" (UniqueName: \"kubernetes.io/projected/9bea213e-5ff8-447d-98db-fa0ec51cda10-kube-api-access-bx97c\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:33.785564 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.785544    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7d93f72-6fc0-469a-8c16-9feed09bda86-konnectivity-ca\") pod \"konnectivity-agent-spbjm\" (UID: \"c7d93f72-6fc0-469a-8c16-9feed09bda86\") " pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:56:33.785630 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.785576    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-cni-multus\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.785630 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.785592    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-etc-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.785630 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.785607    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-log-socket\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.785630 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.785622    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb94c55-ff22-4e24-99da-047c765a8598-cert\") pod \"ingress-canary-jcqwk\" (UID: \"9fb94c55-ff22-4e24-99da-047c765a8598\") " pod="openshift-ingress-canary/ingress-canary-jcqwk"
Nov 02 14:56:33.785766 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.785660    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:33.785766 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.785695    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:33.785766 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.785724    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-registration-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.785857 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.785772    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:33.785893 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.785859    4891 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:56:33.785923 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.785894    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:33.785923 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.785921    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert podName:fb39336d-67c0-4cf2-aaff-c873d361f9b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.285907551 +0000 UTC m=+1.878524348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mf5tv" (UID: "fb39336d-67c0-4cf2-aaff-c873d361f9b2") : secret "networking-console-plugin-cert" not found
Nov 02 14:56:33.786002 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.785956    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7d93f72-6fc0-469a-8c16-9feed09bda86-konnectivity-ca\") pod \"konnectivity-agent-spbjm\" (UID: \"c7d93f72-6fc0-469a-8c16-9feed09bda86\") " pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:56:33.786002 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.785993    4891 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:56:33.786069 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.785993    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-conf-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.786069 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.785997    4891 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:33.786069 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.786021    4891 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74fd4fbb69-thwc5: secret "image-registry-tls" not found
Nov 02 14:56:33.786069 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.785993    4891 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:33.786188 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.786073    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls podName:b639a9d2-f451-4cb1-a31f-c3de10b8d3c1 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.286056498 +0000 UTC m=+1.878673284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls") pod "dns-default-f6fgv" (UID: "b639a9d2-f451-4cb1-a31f-c3de10b8d3c1") : secret "dns-default-metrics-tls" not found
Nov 02 14:56:33.786188 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786083    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb94c55-ff22-4e24-99da-047c765a8598-cert\") pod \"ingress-canary-jcqwk\" (UID: \"9fb94c55-ff22-4e24-99da-047c765a8598\") " pod="openshift-ingress-canary/ingress-canary-jcqwk"
Nov 02 14:56:33.786188 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.786093    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls podName:8d2ae30e-0191-41ff-85da-dedf17ad27b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.286082995 +0000 UTC m=+1.878699776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls") pod "image-registry-74fd4fbb69-thwc5" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2") : secret "image-registry-tls" not found
Nov 02 14:56:33.786188 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.786109    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls podName:885145c5-6077-4435-89ce-73af1e6a5600 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.286100443 +0000 UTC m=+1.878717220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-q2t8x" (UID: "885145c5-6077-4435-89ce-73af1e6a5600") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:33.786188 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786127    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-data-volume\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:33.786188 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786149    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psj4b\" (UniqueName: \"kubernetes.io/projected/1bcba632-1434-4533-951f-84b8bba1f0d8-kube-api-access-psj4b\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:33.786420 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786199    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-data-volume\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:33.786420 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786223    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.786420 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786241    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysctl-d\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.786420 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786257    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-stats-auth\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:33.786420 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786273    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-dbus\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:33.786420 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786288    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkkc\" (UniqueName: \"kubernetes.io/projected/9fb94c55-ff22-4e24-99da-047c765a8598-kube-api-access-xvkkc\") pod \"ingress-canary-jcqwk\" (UID: \"9fb94c55-ff22-4e24-99da-047c765a8598\") " pod="openshift-ingress-canary/ingress-canary-jcqwk"
Nov 02 14:56:33.786420 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786360    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.786420 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786364    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg79x\" (UniqueName: \"kubernetes.io/projected/63faa28d-8806-4332-99c4-9f715a0b1a86-kube-api-access-kg79x\") pod \"kas-connection-checker-685895c7c9-64bm4\" (UID: \"63faa28d-8806-4332-99c4-9f715a0b1a86\") " pod="kube-system/kas-connection-checker-685895c7c9-64bm4"
Nov 02 14:56:33.786420 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786396    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysctl-d\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.786420 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786399    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-ovn\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786509    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-stats-auth\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786517    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-cni-bin\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786550    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-socket-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786593    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786616    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-kubernetes\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786632    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npd9z\" (UniqueName: \"kubernetes.io/projected/100e3924-bcfc-4208-84ea-c1ff1656e949-kube-api-access-npd9z\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.786635    4891 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786656    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786675    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-kubernetes\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786681    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovnkube-config\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.786694    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs podName:9bea213e-5ff8-447d-98db-fa0ec51cda10 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.286677749 +0000 UTC m=+1.879294539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs") pod "network-metrics-daemon-rkxt7" (UID: "9bea213e-5ff8-447d-98db-fa0ec51cda10") : secret "metrics-daemon-secret" not found
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786726    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786751    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-node-log\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.786848 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786847    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovnkube-config\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786859    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovn-node-metrics-cert\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786895    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flmc9\" (UniqueName: \"kubernetes.io/projected/0290bc25-3e34-4d43-9215-f15cfc44f686-kube-api-access-flmc9\") pod \"kas-connection-checker-685895c7c9-xx5tn\" (UID: \"0290bc25-3e34-4d43-9215-f15cfc44f686\") " pod="kube-system/kas-connection-checker-685895c7c9-xx5tn"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786923    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786955    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.787008    4891 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.786985    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-var-lib-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787039    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-kubelet-config\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787043    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovn-node-metrics-cert\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.787066    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls podName:7ff63dfc-17d4-4efc-a23a-d2e46a6885fc nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.287049879 +0000 UTC m=+1.879666670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vhx5m" (UID: "7ff63dfc-17d4-4efc-a23a-d2e46a6885fc") : secret "samples-operator-tls" not found
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787103    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-multus-certs\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787131    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-system-cni-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787187    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-system-cni-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787211    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787228    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-netns\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787245    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787263    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-slash\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787287    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-sys-fs\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787295    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-netns\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787397    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-cni-bin\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787419    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-etc-kubernetes\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.787420    4891 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.787433    4891 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcff7b7b8-bck5q: secret "image-registry-tls" not found
Nov 02 14:56:33.787474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787462    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cnibin\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787497    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-cnibin\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787502    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovnkube-script-lib\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.787517    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls podName:fac285e0-f60a-4f53-a7e8-dee42bb83a0a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.287504117 +0000 UTC m=+1.880120893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls") pod "image-registry-dcff7b7b8-bck5q" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a") : secret "image-registry-tls" not found
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787544    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7d93f72-6fc0-469a-8c16-9feed09bda86-agent-certs\") pod \"konnectivity-agent-spbjm\" (UID: \"c7d93f72-6fc0-469a-8c16-9feed09bda86\") " pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787562    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-cnibin\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787586    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-cni-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787606    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-kubelet\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787634    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787639    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-cnibin\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787664    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-cni-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787666    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-run\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787691    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-ovnkube-script-lib\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787713    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-socket-dir-parent\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787734    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-run\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787745    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-hostroot\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787770    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74049a08-892a-4ae2-a5a5-161da6182c1c-hosts-file\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787781    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-socket-dir-parent\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787786    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acede18e-4d8e-47ee-b31f-918658a9ffba-host\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787788    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7d93f72-6fc0-469a-8c16-9feed09bda86-agent-certs\") pod \"konnectivity-agent-spbjm\" (UID: \"c7d93f72-6fc0-469a-8c16-9feed09bda86\") " pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787833    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74049a08-892a-4ae2-a5a5-161da6182c1c-hosts-file\") pod \"node-resolver-cllms\" (UID: \"74049a08-892a-4ae2-a5a5-161da6182c1c\") " pod="openshift-dns/node-resolver-cllms"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787853    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-crio-socket\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787878    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acede18e-4d8e-47ee-b31f-918658a9ffba-host\") pod \"node-ca-9htnf\" (UID: \"acede18e-4d8e-47ee-b31f-918658a9ffba\") " pod="openshift-image-registry/node-ca-9htnf"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787875    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-kubelet\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787915    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787937    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.787971    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-var-lib-kubelet\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788048    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-var-lib-kubelet\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788070    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-device-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788088    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2pbg\" (UniqueName: \"kubernetes.io/projected/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-kube-api-access-k2pbg\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788107    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fb8\" (UniqueName: \"kubernetes.io/projected/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-api-access-s7fb8\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788125    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788173    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bb718bc-429a-4875-bd8d-d52dbba0b252-host-slash\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788176    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-original-pull-secret\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788202    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysconfig\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788237    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bb718bc-429a-4875-bd8d-d52dbba0b252-host-slash\") pod \"iptables-alerter-whl44\" (UID: \"9bb718bc-429a-4875-bd8d-d52dbba0b252\") " pod="openshift-network-operator/iptables-alerter-whl44"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788245    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysconfig\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788334    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-systemd\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788357    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-lib-modules\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788374    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-host\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.788370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788389    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysctl-conf\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788386    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-systemd\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788412    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-cni-netd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788434    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-env-overrides\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788445    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-host\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788450    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-daemon-config\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788464    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-lib-modules\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788488    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-systemd-units\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788516    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-sysctl-conf\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788533    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-systemd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788575    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-modprobe-d\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788604    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f2ebec5-9906-438f-b2ce-010345fa49ba-env-overrides\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788630    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-k8s-cni-cncf-io\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788657    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-default-certificate\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788661    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-etc-modprobe-d\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788682    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-run-netns\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788702    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-k8s-cni-cncf-io\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788707    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-run-ovn-kubernetes\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788731    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmfrd\" (UniqueName: \"kubernetes.io/projected/6f2ebec5-9906-438f-b2ce-010345fa49ba-kube-api-access-rmfrd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788747    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788761    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-system-cni-dir\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788771    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-daemon-config\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788781    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-sys\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788822    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac-system-cni-dir\") pod \"multus-additional-cni-plugins-596b4\" (UID: \"2e22bc1a-6a91-4b05-94ce-e3f11ccb0eac\") " pod="openshift-multus/multus-additional-cni-plugins-596b4"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788885    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19ce39d1-e591-4af6-81e4-63d7d7b255c4-sys\") pod \"tuned-5lhfv\" (UID: \"19ce39d1-e591-4af6-81e4-63d7d7b255c4\") " pod="openshift-cluster-node-tuning-operator/tuned-5lhfv"
Nov 02 14:56:33.789598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.788909    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-default-certificate\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:33.847126 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.847098    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npd9z\" (UniqueName: \"kubernetes.io/projected/100e3924-bcfc-4208-84ea-c1ff1656e949-kube-api-access-npd9z\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.847435 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.847416    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flmc9\" (UniqueName: \"kubernetes.io/projected/0290bc25-3e34-4d43-9215-f15cfc44f686-kube-api-access-flmc9\") pod \"kas-connection-checker-685895c7c9-xx5tn\" (UID: \"0290bc25-3e34-4d43-9215-f15cfc44f686\") " pod="kube-system/kas-connection-checker-685895c7c9-xx5tn"
Nov 02 14:56:33.847677 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.847660    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psj4b\" (UniqueName: \"kubernetes.io/projected/1bcba632-1434-4533-951f-84b8bba1f0d8-kube-api-access-psj4b\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:33.847926 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.847910    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkkc\" (UniqueName: \"kubernetes.io/projected/9fb94c55-ff22-4e24-99da-047c765a8598-kube-api-access-xvkkc\") pod \"ingress-canary-jcqwk\" (UID: \"9fb94c55-ff22-4e24-99da-047c765a8598\") " pod="openshift-ingress-canary/ingress-canary-jcqwk"
Nov 02 14:56:33.848733 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.848715    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg79x\" (UniqueName: \"kubernetes.io/projected/63faa28d-8806-4332-99c4-9f715a0b1a86-kube-api-access-kg79x\") pod \"kas-connection-checker-685895c7c9-64bm4\" (UID: \"63faa28d-8806-4332-99c4-9f715a0b1a86\") " pod="kube-system/kas-connection-checker-685895c7c9-64bm4"
Nov 02 14:56:33.848795 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.848720    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2pbg\" (UniqueName: \"kubernetes.io/projected/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-kube-api-access-k2pbg\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.849143 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.849128    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmfrd\" (UniqueName: \"kubernetes.io/projected/6f2ebec5-9906-438f-b2ce-010345fa49ba-kube-api-access-rmfrd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.852506 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.852491    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fb8\" (UniqueName: \"kubernetes.io/projected/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-api-access-s7fb8\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:33.889718 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889695    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:33.889814 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889721    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-slash\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.889814 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889736    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-sys-fs\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.889814 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889752    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-cni-bin\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.889814 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889767    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-etc-kubernetes\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.889814 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889797    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-slash\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.889814 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889811    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-etc-kubernetes\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.890002 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.889834    4891 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:56:33.890002 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889856    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-kubelet\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.890002 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889838    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-cni-bin\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.890002 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889886    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:33.890002 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889833    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-sys-fs\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.890002 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889915    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-kubelet\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.890002 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.889925    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.389908326 +0000 UTC m=+1.982525125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : configmap "kube-rbac-proxy" not found
Nov 02 14:56:33.890002 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889942    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-hostroot\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.890002 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889954    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-hostroot\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.890002 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.889955    4891 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:33.890002 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.889973    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-crio-socket\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:33.890002 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.889996    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.389983767 +0000 UTC m=+1.982600568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890011    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-crio-socket\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890014    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-kubelet\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890036    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890054    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-device-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890081    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890110    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890115    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890114    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-cni-netd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890139    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-systemd-units\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890151    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-device-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890153    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-systemd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890170    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-systemd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890180    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-run-netns\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890194    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-systemd-units\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890217    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-run-netns\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890219    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-kubelet\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890204    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-run-ovn-kubernetes\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890255    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-run-ovn-kubernetes\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890260    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890291    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-cni-multus\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890324    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-etc-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890332    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-var-lib-cni-multus\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890337    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890291    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-cni-netd\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890339    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-log-socket\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890362    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-log-socket\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890367    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-etc-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.890385 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890396    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-registration-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890422    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-conf-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890441    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-dbus\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890456    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-ovn\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890472    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-cni-bin\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890476    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-multus-conf-dir\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890487    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-socket-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890511    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-registration-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890518    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890525    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-cni-bin\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890528    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-dbus\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890513    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-run-ovn\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890546    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.890568    4891 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890559    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49e40d7f-ab6a-4817-ba3e-3e2ab361b454-socket-dir\") pod \"aws-ebs-csi-driver-node-sjp8t\" (UID: \"49e40d7f-ab6a-4817-ba3e-3e2ab361b454\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjp8t"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890584    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-node-log\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.890593    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.39058492 +0000 UTC m=+1.983201741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : configmap references non-existent config key: service-ca.crt
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:33.890620    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.39060683 +0000 UTC m=+1.983223622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : secret "router-metrics-certs-default" not found
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890635    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-node-log\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890668    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890686    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-var-lib-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890702    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-kubelet-config\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890718    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-multus-certs\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890754    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-var-lib-openvswitch\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890776    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41cff577-8a79-4b9e-89c8-a93e1ecf4ff0-kubelet-config\") pod \"global-pull-secret-syncer-nx6px\" (UID: \"41cff577-8a79-4b9e-89c8-a93e1ecf4ff0\") " pod="kube-system/global-pull-secret-syncer-nx6px"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890784    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2ebec5-9906-438f-b2ce-010345fa49ba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8822k\" (UID: \"6f2ebec5-9906-438f-b2ce-010345fa49ba\") " pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:33.891168 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.890788    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/100e3924-bcfc-4208-84ea-c1ff1656e949-host-run-multus-certs\") pod \"multus-w5w2m\" (UID: \"100e3924-bcfc-4208-84ea-c1ff1656e949\") " pod="openshift-multus/multus-w5w2m"
Nov 02 14:56:33.923133 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:33.923110    4891 scope.go:122] "RemoveContainer" containerID="e65d58d2a1b20beec085715ef00a07f1555863222f9e3afe972c3aa920641f19"
Nov 02 14:56:34.294978 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.294945    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:34.295131 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295055    4891 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 14:56:34.295131 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.295095    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:34.295131 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295111    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs podName:9bea213e-5ff8-447d-98db-fa0ec51cda10 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:35.295098142 +0000 UTC m=+2.887714936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs") pod "network-metrics-daemon-rkxt7" (UID: "9bea213e-5ff8-447d-98db-fa0ec51cda10") : secret "metrics-daemon-secret" not found
Nov 02 14:56:34.295260 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.295148    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:34.295260 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295166    4891 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:56:34.295260 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295207    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls podName:7ff63dfc-17d4-4efc-a23a-d2e46a6885fc nodeName:}" failed. No retries permitted until 2026-11-02 14:56:35.295195375 +0000 UTC m=+2.887812174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vhx5m" (UID: "7ff63dfc-17d4-4efc-a23a-d2e46a6885fc") : secret "samples-operator-tls" not found
Nov 02 14:56:34.295260 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295235    4891 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:34.295260 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295244    4891 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcff7b7b8-bck5q: secret "image-registry-tls" not found
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.295265    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295272    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls podName:fac285e0-f60a-4f53-a7e8-dee42bb83a0a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:35.295264338 +0000 UTC m=+2.887881149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls") pod "image-registry-dcff7b7b8-bck5q" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a") : secret "image-registry-tls" not found
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.295293    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295326    4891 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295356    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls podName:b639a9d2-f451-4cb1-a31f-c3de10b8d3c1 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:35.295348879 +0000 UTC m=+2.887965673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls") pod "dns-default-f6fgv" (UID: "b639a9d2-f451-4cb1-a31f-c3de10b8d3c1") : secret "dns-default-metrics-tls" not found
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295360    4891 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.295327    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295367    4891 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74fd4fbb69-thwc5: secret "image-registry-tls" not found
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.295381    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295386    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls podName:8d2ae30e-0191-41ff-85da-dedf17ad27b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:35.295379876 +0000 UTC m=+2.887996686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls") pod "image-registry-74fd4fbb69-thwc5" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2") : secret "image-registry-tls" not found
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295390    4891 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295422    4891 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295436    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert podName:fb39336d-67c0-4cf2-aaff-c873d361f9b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:35.295424701 +0000 UTC m=+2.888041492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mf5tv" (UID: "fb39336d-67c0-4cf2-aaff-c873d361f9b2") : secret "networking-console-plugin-cert" not found
Nov 02 14:56:34.295448 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.295450    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls podName:885145c5-6077-4435-89ce-73af1e6a5600 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:35.295443202 +0000 UTC m=+2.888059978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-q2t8x" (UID: "885145c5-6077-4435-89ce-73af1e6a5600") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:34.396274 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.396246    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:34.396274 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.396277    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:34.396467 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.396328    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:34.396467 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.396384    4891 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:56:34.396467 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.396407    4891 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:56:34.396467 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.396425    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:35.396414803 +0000 UTC m=+2.989031593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : configmap "kube-rbac-proxy" not found
Nov 02 14:56:34.396467 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.396437    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:35.396431283 +0000 UTC m=+2.989048060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : configmap references non-existent config key: service-ca.crt
Nov 02 14:56:34.396467 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.396458    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:34.396678 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.396502    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:35.396495604 +0000 UTC m=+2.989112380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : secret "router-metrics-certs-default" not found
Nov 02 14:56:34.396678 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.396530    4891 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:34.396678 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.396561    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:35.396551081 +0000 UTC m=+2.989167888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:34.569095 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.569030    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 14:56:34.569494 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.569407    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/1.log"
Nov 02 14:56:34.569494 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.569435    4891 generic.go:356] "Generic (PLEG): container finished" podID="bf04dbe2-4354-4849-959c-33002c7ece68" containerID="dc4e4988fdc3b39ac1a91bc7bb41b106e75e402ce3ed9800ea56caf13a72d9f7" exitCode=255
Nov 02 14:56:34.569557 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.569511    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv" event={"ID":"bf04dbe2-4354-4849-959c-33002c7ece68","Type":"ContainerDied","Data":"dc4e4988fdc3b39ac1a91bc7bb41b106e75e402ce3ed9800ea56caf13a72d9f7"}
Nov 02 14:56:34.569592 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.569572    4891 scope.go:122] "RemoveContainer" containerID="e65d58d2a1b20beec085715ef00a07f1555863222f9e3afe972c3aa920641f19"
Nov 02 14:56:34.570720 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:34.570695    4891 scope.go:122] "RemoveContainer" containerID="dc4e4988fdc3b39ac1a91bc7bb41b106e75e402ce3ed9800ea56caf13a72d9f7"
Nov 02 14:56:34.571116 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:34.571095    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rtsrv_openshift-console-operator(bf04dbe2-4354-4849-959c-33002c7ece68)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv" podUID="bf04dbe2-4354-4849-959c-33002c7ece68"
Nov 02 14:56:35.003960 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.003878    4891 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:35.304433 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.304399    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:35.304601 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.304443    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:35.304601 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304544    4891 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:56:35.304601 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.304570    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:35.304753 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304615    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert podName:fb39336d-67c0-4cf2-aaff-c873d361f9b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.304597835 +0000 UTC m=+4.897214627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mf5tv" (UID: "fb39336d-67c0-4cf2-aaff-c873d361f9b2") : secret "networking-console-plugin-cert" not found
Nov 02 14:56:35.304753 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304544    4891 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:56:35.304753 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.304636    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:35.304753 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304658    4891 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:35.304753 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304672    4891 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74fd4fbb69-thwc5: secret "image-registry-tls" not found
Nov 02 14:56:35.304753 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304687    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls podName:b639a9d2-f451-4cb1-a31f-c3de10b8d3c1 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.30466986 +0000 UTC m=+4.897286654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls") pod "dns-default-f6fgv" (UID: "b639a9d2-f451-4cb1-a31f-c3de10b8d3c1") : secret "dns-default-metrics-tls" not found
Nov 02 14:56:35.304753 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304705    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls podName:8d2ae30e-0191-41ff-85da-dedf17ad27b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.304694817 +0000 UTC m=+4.897311598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls") pod "image-registry-74fd4fbb69-thwc5" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2") : secret "image-registry-tls" not found
Nov 02 14:56:35.304753 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.304721    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:35.304753 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304726    4891 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:35.305135 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304763    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls podName:885145c5-6077-4435-89ce-73af1e6a5600 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.304751085 +0000 UTC m=+4.897367876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-q2t8x" (UID: "885145c5-6077-4435-89ce-73af1e6a5600") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:35.305135 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.304781    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:35.305135 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304787    4891 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 14:56:35.305135 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.304823    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:35.305135 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304820    4891 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:56:35.305135 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304877    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs podName:9bea213e-5ff8-447d-98db-fa0ec51cda10 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.304865691 +0000 UTC m=+4.897482468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs") pod "network-metrics-daemon-rkxt7" (UID: "9bea213e-5ff8-447d-98db-fa0ec51cda10") : secret "metrics-daemon-secret" not found
Nov 02 14:56:35.305135 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304904    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls podName:7ff63dfc-17d4-4efc-a23a-d2e46a6885fc nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.304894061 +0000 UTC m=+4.897510837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vhx5m" (UID: "7ff63dfc-17d4-4efc-a23a-d2e46a6885fc") : secret "samples-operator-tls" not found
Nov 02 14:56:35.305135 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304909    4891 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:35.305135 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304927    4891 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcff7b7b8-bck5q: secret "image-registry-tls" not found
Nov 02 14:56:35.305135 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.304994    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls podName:fac285e0-f60a-4f53-a7e8-dee42bb83a0a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.304982965 +0000 UTC m=+4.897599756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls") pod "image-registry-dcff7b7b8-bck5q" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a") : secret "image-registry-tls" not found
Nov 02 14:56:35.405737 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.405702    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:35.405737 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.405734    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:35.405937 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.405842    4891 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:56:35.405937 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.405844    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.40583132 +0000 UTC m=+4.998448109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : configmap references non-existent config key: service-ca.crt
Nov 02 14:56:35.405937 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.405905    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:35.406032 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.405941    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:35.406032 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.405958    4891 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:56:35.406032 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.405990    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.405980151 +0000 UTC m=+4.998596932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : configmap "kube-rbac-proxy" not found
Nov 02 14:56:35.406032 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.406011    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.406000588 +0000 UTC m=+4.998617372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : secret "router-metrics-certs-default" not found
Nov 02 14:56:35.406032 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.406018    4891 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:35.406176 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.406048    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:37.406036843 +0000 UTC m=+4.998653633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:35.574099 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.574025    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 14:56:35.574484 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:35.574355    4891 scope.go:122] "RemoveContainer" containerID="dc4e4988fdc3b39ac1a91bc7bb41b106e75e402ce3ed9800ea56caf13a72d9f7"
Nov 02 14:56:35.574544 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:35.574526    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rtsrv_openshift-console-operator(bf04dbe2-4354-4849-959c-33002c7ece68)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv" podUID="bf04dbe2-4354-4849-959c-33002c7ece68"
Nov 02 14:56:36.554167 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:36.554140    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:36.918669 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:36.918590    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:56:37.323647 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.323610    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:37.323801 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.323675    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:37.323801 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.323716    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:37.323801 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323755    4891 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 14:56:37.323801 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323770    4891 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:56:37.323939 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323826    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls podName:7ff63dfc-17d4-4efc-a23a-d2e46a6885fc nodeName:}" failed. No retries permitted until 2026-11-02 14:56:41.323809554 +0000 UTC m=+8.916426348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vhx5m" (UID: "7ff63dfc-17d4-4efc-a23a-d2e46a6885fc") : secret "samples-operator-tls" not found
Nov 02 14:56:37.323939 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323842    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs podName:9bea213e-5ff8-447d-98db-fa0ec51cda10 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:41.323835534 +0000 UTC m=+8.916452311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs") pod "network-metrics-daemon-rkxt7" (UID: "9bea213e-5ff8-447d-98db-fa0ec51cda10") : secret "metrics-daemon-secret" not found
Nov 02 14:56:37.323939 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323841    4891 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:37.323939 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.323853    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:37.323939 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323859    4891 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dcff7b7b8-bck5q: secret "image-registry-tls" not found
Nov 02 14:56:37.323939 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.323874    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:37.323939 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323899    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls podName:fac285e0-f60a-4f53-a7e8-dee42bb83a0a nodeName:}" failed. No retries permitted until 2026-11-02 14:56:41.323885749 +0000 UTC m=+8.916502526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls") pod "image-registry-dcff7b7b8-bck5q" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a") : secret "image-registry-tls" not found
Nov 02 14:56:37.323939 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323922    4891 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:56:37.323939 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323938    4891 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:56:37.324218 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323950    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert podName:fb39336d-67c0-4cf2-aaff-c873d361f9b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:41.3239397 +0000 UTC m=+8.916556509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mf5tv" (UID: "fb39336d-67c0-4cf2-aaff-c873d361f9b2") : secret "networking-console-plugin-cert" not found
Nov 02 14:56:37.324218 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323987    4891 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 14:56:37.324218 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323991    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls podName:b639a9d2-f451-4cb1-a31f-c3de10b8d3c1 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:41.323976949 +0000 UTC m=+8.916593730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls") pod "dns-default-f6fgv" (UID: "b639a9d2-f451-4cb1-a31f-c3de10b8d3c1") : secret "dns-default-metrics-tls" not found
Nov 02 14:56:37.324218 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.323995    4891 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74fd4fbb69-thwc5: secret "image-registry-tls" not found
Nov 02 14:56:37.324218 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.323940    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:37.324218 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.324018    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls podName:8d2ae30e-0191-41ff-85da-dedf17ad27b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:41.324011703 +0000 UTC m=+8.916628480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls") pod "image-registry-74fd4fbb69-thwc5" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2") : secret "image-registry-tls" not found
Nov 02 14:56:37.324218 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.324031    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:37.324218 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.324100    4891 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:37.324218 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.324126    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls podName:885145c5-6077-4435-89ce-73af1e6a5600 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:41.324120311 +0000 UTC m=+8.916737087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-q2t8x" (UID: "885145c5-6077-4435-89ce-73af1e6a5600") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:37.424637 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.424598    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:37.424814 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.424711    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:37.424814 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.424729    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:37.424814 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.424756    4891 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:37.424814 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.424767    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:37.424991 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.424828    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:41.424809543 +0000 UTC m=+9.017426342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:37.424991 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.424842    4891 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:56:37.424991 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.424870    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:41.424857762 +0000 UTC m=+9.017474581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : configmap references non-existent config key: service-ca.crt
Nov 02 14:56:37.424991 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.424891    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:41.424882936 +0000 UTC m=+9.017499712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : secret "router-metrics-certs-default" not found
Nov 02 14:56:37.424991 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.424893    4891 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:56:37.424991 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.424926    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:41.424911482 +0000 UTC m=+9.017528263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : configmap "kube-rbac-proxy" not found
Nov 02 14:56:37.654912 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.654830    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cllms_74049a08-892a-4ae2-a5a5-161da6182c1c/dns-node-resolver/0.log"
Nov 02 14:56:37.673822 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.673800    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:37.674162 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:37.674149    4891 scope.go:122] "RemoveContainer" containerID="dc4e4988fdc3b39ac1a91bc7bb41b106e75e402ce3ed9800ea56caf13a72d9f7"
Nov 02 14:56:37.674353 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:37.674335    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rtsrv_openshift-console-operator(bf04dbe2-4354-4849-959c-33002c7ece68)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv" podUID="bf04dbe2-4354-4849-959c-33002c7ece68"
Nov 02 14:56:38.447105 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:38.447079    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9htnf_acede18e-4d8e-47ee-b31f-918658a9ffba/node-ca/0.log"
Nov 02 14:56:38.597456 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:38.597426    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:38.597677 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:38.597661    4891 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:56:39.446870 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:39.446835    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jcqwk_9fb94c55-ff22-4e24-99da-047c765a8598/serve-healthcheck-canary/0.log"
Nov 02 14:56:40.247785 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:40.247751    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-lz25d_234a4a99-5491-44ef-b58c-3e4ab32584e5/kube-storage-version-migrator-operator/0.log"
Nov 02 14:56:41.362565 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.362518    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.362570    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.362599    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.362627    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.362669    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.362673    4891 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.362735    4891 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.362773    4891 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.362802    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls podName:b639a9d2-f451-4cb1-a31f-c3de10b8d3c1 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:49.362780511 +0000 UTC m=+16.955397300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls") pod "dns-default-f6fgv" (UID: "b639a9d2-f451-4cb1-a31f-c3de10b8d3c1") : secret "dns-default-metrics-tls" not found
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.362827    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert podName:fb39336d-67c0-4cf2-aaff-c873d361f9b2 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:49.3628149 +0000 UTC m=+16.955431677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mf5tv" (UID: "fb39336d-67c0-4cf2-aaff-c873d361f9b2") : secret "networking-console-plugin-cert" not found
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.362844    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls podName:885145c5-6077-4435-89ce-73af1e6a5600 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:49.362834398 +0000 UTC m=+16.955451175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-q2t8x" (UID: "885145c5-6077-4435-89ce-73af1e6a5600") : secret "cluster-monitoring-operator-tls" not found
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.362892    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.362939    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.363004    4891 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.363040    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls podName:7ff63dfc-17d4-4efc-a23a-d2e46a6885fc nodeName:}" failed. No retries permitted until 2026-11-02 14:56:49.363028161 +0000 UTC m=+16.955644963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vhx5m" (UID: "7ff63dfc-17d4-4efc-a23a-d2e46a6885fc") : secret "samples-operator-tls" not found
Nov 02 14:56:41.363071 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.363069    4891 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 14:56:41.365226 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.365197    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bea213e-5ff8-447d-98db-fa0ec51cda10-metrics-certs\") pod \"network-metrics-daemon-rkxt7\" (UID: \"9bea213e-5ff8-447d-98db-fa0ec51cda10\") " pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:41.365409 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.365389    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"image-registry-dcff7b7b8-bck5q\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") " pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:41.365409 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.365402    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"image-registry-74fd4fbb69-thwc5\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") " pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:41.425888 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.425858    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:41.430925 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.430902    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkxt7"
Nov 02 14:56:41.435818 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.435791    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:41.465366 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.464496    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:41.465366 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.464553    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:41.465366 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.464597    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:41.465366 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.464623    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:41.465366 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.464803    4891 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:41.465366 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.464882    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:49.464863442 +0000 UTC m=+17.057480237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : secret "insights-runtime-extractor-tls" not found
Nov 02 14:56:41.465366 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.464942    4891 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 14:56:41.465366 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.464978    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:49.464965827 +0000 UTC m=+17.057582622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : secret "router-metrics-certs-default" not found
Nov 02 14:56:41.465366 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.465043    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle podName:1bcba632-1434-4533-951f-84b8bba1f0d8 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:49.465033903 +0000 UTC m=+17.057650724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle") pod "router-default-684675bd5f-zwwbp" (UID: "1bcba632-1434-4533-951f-84b8bba1f0d8") : configmap references non-existent config key: service-ca.crt
Nov 02 14:56:41.465366 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.465082    4891 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 14:56:41.465366 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:56:41.465112    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm podName:b7a4115d-9a1a-4ed3-ba75-05c621c1fa63 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:49.465102006 +0000 UTC m=+17.057718798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtxzd" (UID: "b7a4115d-9a1a-4ed3-ba75-05c621c1fa63") : configmap "kube-rbac-proxy" not found
Nov 02 14:56:41.900702 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.900665    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rkxt7"]
Nov 02 14:56:41.904934 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:41.904906    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bea213e_5ff8_447d_98db_fa0ec51cda10.slice/crio-2ce482e390c5b5a12fcc86807dd5559290f47e01f65d7e47f45115ad0f429a32 WatchSource:0}: Error finding container 2ce482e390c5b5a12fcc86807dd5559290f47e01f65d7e47f45115ad0f429a32: Status 404 returned error can't find the container with id 2ce482e390c5b5a12fcc86807dd5559290f47e01f65d7e47f45115ad0f429a32
Nov 02 14:56:41.906912 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.906895    4891 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:56:41.944664 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:41.944634    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2ae30e_0191_41ff_85da_dedf17ad27b2.slice/crio-33da889bfc3403307eb6f2afeb4fffeca757b1f4c60fee8c7737afa3ef247f74 WatchSource:0}: Error finding container 33da889bfc3403307eb6f2afeb4fffeca757b1f4c60fee8c7737afa3ef247f74: Status 404 returned error can't find the container with id 33da889bfc3403307eb6f2afeb4fffeca757b1f4c60fee8c7737afa3ef247f74
Nov 02 14:56:41.950219 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.950193    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74fd4fbb69-thwc5"]
Nov 02 14:56:41.957163 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:41.957142    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dcff7b7b8-bck5q"]
Nov 02 14:56:41.960979 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:41.960958    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac285e0_f60a_4f53_a7e8_dee42bb83a0a.slice/crio-d67cfbb786aa3abb4518fabf0aac1db65d43a53c8784a03c7524930433188639 WatchSource:0}: Error finding container d67cfbb786aa3abb4518fabf0aac1db65d43a53c8784a03c7524930433188639: Status 404 returned error can't find the container with id d67cfbb786aa3abb4518fabf0aac1db65d43a53c8784a03c7524930433188639
Nov 02 14:56:42.596780 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:42.596746    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q" event={"ID":"fac285e0-f60a-4f53-a7e8-dee42bb83a0a","Type":"ContainerStarted","Data":"1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321"}
Nov 02 14:56:42.597245 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:42.596804    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q" event={"ID":"fac285e0-f60a-4f53-a7e8-dee42bb83a0a","Type":"ContainerStarted","Data":"d67cfbb786aa3abb4518fabf0aac1db65d43a53c8784a03c7524930433188639"}
Nov 02 14:56:42.597984 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:42.597959    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rkxt7" event={"ID":"9bea213e-5ff8-447d-98db-fa0ec51cda10","Type":"ContainerStarted","Data":"2ce482e390c5b5a12fcc86807dd5559290f47e01f65d7e47f45115ad0f429a32"}
Nov 02 14:56:42.599491 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:42.599469    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5" event={"ID":"8d2ae30e-0191-41ff-85da-dedf17ad27b2","Type":"ContainerStarted","Data":"b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4"}
Nov 02 14:56:42.599588 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:42.599501    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5" event={"ID":"8d2ae30e-0191-41ff-85da-dedf17ad27b2","Type":"ContainerStarted","Data":"33da889bfc3403307eb6f2afeb4fffeca757b1f4c60fee8c7737afa3ef247f74"}
Nov 02 14:56:43.073628 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:43.073596    4891 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-27.us-west-1.compute.internal" event="NodeReady"
Nov 02 14:56:43.097971 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:43.097942    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:56:43.604084 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:43.603988    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rkxt7" event={"ID":"9bea213e-5ff8-447d-98db-fa0ec51cda10","Type":"ContainerStarted","Data":"9a4f44f6c00428567bcfe4b1627a43224b5f259c5bfe731460ed76c5cbacc8d2"}
Nov 02 14:56:43.604084 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:43.604032    4891 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:56:43.604084 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:43.604047    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rkxt7" event={"ID":"9bea213e-5ff8-447d-98db-fa0ec51cda10","Type":"ContainerStarted","Data":"8dce6a0f420b67c9c7e1608506d77366b8d4d36536c27b18e43509bdbfc2be28"}
Nov 02 14:56:45.514813 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:45.514778    4891 scope.go:122] "RemoveContainer" containerID="dc4e4988fdc3b39ac1a91bc7bb41b106e75e402ce3ed9800ea56caf13a72d9f7"
Nov 02 14:56:45.614663 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:45.614631    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 14:56:45.614820 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:45.614713    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv" event={"ID":"bf04dbe2-4354-4849-959c-33002c7ece68","Type":"ContainerStarted","Data":"39f6f08cf922759b3fea9c85f9be94f8e94ece93c8c37a881c21aa0944b9f578"}
Nov 02 14:56:45.615009 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:45.614988    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:45.615908 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:45.615885    4891 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-rtsrv container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body=
Nov 02 14:56:45.616003 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:45.615928    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv" podUID="bf04dbe2-4354-4849-959c-33002c7ece68" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused"
Nov 02 14:56:45.770152 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:45.770121    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:56:46.622754 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:46.622727    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rtsrv"
Nov 02 14:56:49.441657 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.441612    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:49.442151 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.441663    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:49.442151 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.441694    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:49.442151 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.441758    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:49.444267 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.444238    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff63dfc-17d4-4efc-a23a-d2e46a6885fc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vhx5m\" (UID: \"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:49.444392 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.444271    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b639a9d2-f451-4cb1-a31f-c3de10b8d3c1-metrics-tls\") pod \"dns-default-f6fgv\" (UID: \"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1\") " pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:49.444392 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.444301    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb39336d-67c0-4cf2-aaff-c873d361f9b2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mf5tv\" (UID: \"fb39336d-67c0-4cf2-aaff-c873d361f9b2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:49.444392 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.444382    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/885145c5-6077-4435-89ce-73af1e6a5600-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-q2t8x\" (UID: \"885145c5-6077-4435-89ce-73af1e6a5600\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:49.524520 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.524479    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"
Nov 02 14:56:49.525565 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.525545    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"
Nov 02 14:56:49.526609 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.526596    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"
Nov 02 14:56:49.536289 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.536252    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:49.543094 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.543062    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:49.543216 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.543106    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:49.543216 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.543136    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:49.543216 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.543170    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:49.545344 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.544199    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:49.545344 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.544277    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bcba632-1434-4533-951f-84b8bba1f0d8-service-ca-bundle\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:49.546207 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.546182    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcba632-1434-4533-951f-84b8bba1f0d8-metrics-certs\") pod \"router-default-684675bd5f-zwwbp\" (UID: \"1bcba632-1434-4533-951f-84b8bba1f0d8\") " pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:49.546843 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.546809    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7a4115d-9a1a-4ed3-ba75-05c621c1fa63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtxzd\" (UID: \"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63\") " pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:49.824200 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.824158    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:49.836406 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.836376    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtxzd"
Nov 02 14:56:49.912557 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.912528    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m"]
Nov 02 14:56:49.965890 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.965860    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-mf5tv"]
Nov 02 14:56:49.969578 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:49.969548    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x"]
Nov 02 14:56:49.971371 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:49.971346    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb39336d_67c0_4cf2_aaff_c873d361f9b2.slice/crio-8c9b458aeda31d558a300490cc7fd3c5a1695555e7add7b927fa0a1cac03f950 WatchSource:0}: Error finding container 8c9b458aeda31d558a300490cc7fd3c5a1695555e7add7b927fa0a1cac03f950: Status 404 returned error can't find the container with id 8c9b458aeda31d558a300490cc7fd3c5a1695555e7add7b927fa0a1cac03f950
Nov 02 14:56:50.005380 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:50.005351    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885145c5_6077_4435_89ce_73af1e6a5600.slice/crio-1da2a4503053bcffd4ad4cabf3ca5bf67b686fc5741a480a50ff8cb0010b4f5f WatchSource:0}: Error finding container 1da2a4503053bcffd4ad4cabf3ca5bf67b686fc5741a480a50ff8cb0010b4f5f: Status 404 returned error can't find the container with id 1da2a4503053bcffd4ad4cabf3ca5bf67b686fc5741a480a50ff8cb0010b4f5f
Nov 02 14:56:50.015483 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.015458    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f6fgv"]
Nov 02 14:56:50.017592 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:50.017565    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb639a9d2_f451_4cb1_a31f_c3de10b8d3c1.slice/crio-c3155f8667efd72029fa15e4f5094eab8d1e4bf59382a523b9b608633f589c2f WatchSource:0}: Error finding container c3155f8667efd72029fa15e4f5094eab8d1e4bf59382a523b9b608633f589c2f: Status 404 returned error can't find the container with id c3155f8667efd72029fa15e4f5094eab8d1e4bf59382a523b9b608633f589c2f
Nov 02 14:56:50.201733 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.201646    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-684675bd5f-zwwbp"]
Nov 02 14:56:50.205076 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:50.205050    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcba632_1434_4533_951f_84b8bba1f0d8.slice/crio-af7e2b28bb9d1ff4820b8e686f88901c367573d1b4039af6b66c46cea0efb068 WatchSource:0}: Error finding container af7e2b28bb9d1ff4820b8e686f88901c367573d1b4039af6b66c46cea0efb068: Status 404 returned error can't find the container with id af7e2b28bb9d1ff4820b8e686f88901c367573d1b4039af6b66c46cea0efb068
Nov 02 14:56:50.229539 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.229511    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dtxzd"]
Nov 02 14:56:50.233597 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:56:50.233569    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a4115d_9a1a_4ed3_ba75_05c621c1fa63.slice/crio-c10e6ae9e4f735b23157bc944b35fe0d37a1db39cda532d4df79bcb65f7c659c WatchSource:0}: Error finding container c10e6ae9e4f735b23157bc944b35fe0d37a1db39cda532d4df79bcb65f7c659c: Status 404 returned error can't find the container with id c10e6ae9e4f735b23157bc944b35fe0d37a1db39cda532d4df79bcb65f7c659c
Nov 02 14:56:50.634728 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.634690    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtxzd" event={"ID":"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63","Type":"ContainerStarted","Data":"c182e7b9f65cc090fbe32a07a04c50e1d2330bf85cc69b53848da6a22515bb3f"}
Nov 02 14:56:50.635149 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.634747    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtxzd" event={"ID":"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63","Type":"ContainerStarted","Data":"c10e6ae9e4f735b23157bc944b35fe0d37a1db39cda532d4df79bcb65f7c659c"}
Nov 02 14:56:50.635883 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.635857    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f6fgv" event={"ID":"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1","Type":"ContainerStarted","Data":"c3155f8667efd72029fa15e4f5094eab8d1e4bf59382a523b9b608633f589c2f"}
Nov 02 14:56:50.636893 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.636871    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv" event={"ID":"fb39336d-67c0-4cf2-aaff-c873d361f9b2","Type":"ContainerStarted","Data":"8c9b458aeda31d558a300490cc7fd3c5a1695555e7add7b927fa0a1cac03f950"}
Nov 02 14:56:50.637942 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.637918    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m" event={"ID":"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc","Type":"ContainerStarted","Data":"aa6e54de4cf5df59f6333ce174a5b608ee6835f44403a70352070c4dcf50c175"}
Nov 02 14:56:50.639329 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.639282    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-684675bd5f-zwwbp" event={"ID":"1bcba632-1434-4533-951f-84b8bba1f0d8","Type":"ContainerStarted","Data":"51f16c3b09ad1dd34861717f442fc1c3036592f93e10ca3a4a6f3efc28e7b21c"}
Nov 02 14:56:50.639413 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.639349    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-684675bd5f-zwwbp" event={"ID":"1bcba632-1434-4533-951f-84b8bba1f0d8","Type":"ContainerStarted","Data":"af7e2b28bb9d1ff4820b8e686f88901c367573d1b4039af6b66c46cea0efb068"}
Nov 02 14:56:50.640514 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.640494    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x" event={"ID":"885145c5-6077-4435-89ce-73af1e6a5600","Type":"ContainerStarted","Data":"1da2a4503053bcffd4ad4cabf3ca5bf67b686fc5741a480a50ff8cb0010b4f5f"}
Nov 02 14:56:50.825283 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.825242    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:50.829245 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:50.829207    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:51.649390 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:51.649325    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:51.651745 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:51.651721    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-684675bd5f-zwwbp"
Nov 02 14:56:51.928921 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:51.922032    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8822k"
Nov 02 14:56:51.928921 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:51.928065    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74fd4fbb69-thwc5"]
Nov 02 14:56:52.662352 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:52.661401    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-mf5tv" event={"ID":"fb39336d-67c0-4cf2-aaff-c873d361f9b2","Type":"ContainerStarted","Data":"e0e48330af8ec64eebcf3977cd07221a40923b6f6b21abdd2241e2569cc0cbe7"}
Nov 02 14:56:52.665146 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:52.665108    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtxzd" event={"ID":"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63","Type":"ContainerStarted","Data":"30f815916c52b3cbeaa5537fb1f7a96009fa1a90079e9382a475c93d7ab9cefa"}
Nov 02 14:56:53.088614 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:53.088468    4891 patch_prober.go:29] interesting pod/image-registry-dcff7b7b8-bck5q container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 14:56:53.088814 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:53.088622    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q" podUID="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:56:53.671472 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:53.671447    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f6fgv" event={"ID":"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1","Type":"ContainerStarted","Data":"cfe904fca6fe5f4c2e5fa19ec5e80a4a914fd58f6b8bed1d96f2421021cf3cdb"}
Nov 02 14:56:53.671802 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:53.671532    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f6fgv" event={"ID":"b639a9d2-f451-4cb1-a31f-c3de10b8d3c1","Type":"ContainerStarted","Data":"b7d74fffe7efdad6b0fe46201fd8acd51051907c72e5ee7c1fd642d53bafe498"}
Nov 02 14:56:53.671802 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:53.671585    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:56:53.673285 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:53.673259    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m" event={"ID":"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc","Type":"ContainerStarted","Data":"f93d3cc6f1c03f019ef56b5a4a5e4302fdec738c3deec90876d7e14b23e62faa"}
Nov 02 14:56:53.673415 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:53.673352    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vhx5m" event={"ID":"7ff63dfc-17d4-4efc-a23a-d2e46a6885fc","Type":"ContainerStarted","Data":"043e3fa81689af1c149c43dac0c1f3255e036c960b3ddd3322ab474ea9cb5436"}
Nov 02 14:56:53.675328 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:53.675291    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-q2t8x" event={"ID":"885145c5-6077-4435-89ce-73af1e6a5600","Type":"ContainerStarted","Data":"e30edfc77192e63ad37a171bbdfdb84542aac16c59c4cb7551427e5f47423143"}
Nov 02 14:56:54.681201 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:56:54.681163    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtxzd" event={"ID":"b7a4115d-9a1a-4ed3-ba75-05c621c1fa63","Type":"ContainerStarted","Data":"f2b9a645157c330bb47e897ae98ee245fa108d9c09d7b94f4a01e8804f616511"}
Nov 02 14:57:00.352009 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:00.351977    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-spbjm"
Nov 02 14:57:01.934342 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:01.934293    4891 patch_prober.go:29] interesting pod/image-registry-74fd4fbb69-thwc5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 14:57:01.934891 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:01.934364    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5" podUID="8d2ae30e-0191-41ff-85da-dedf17ad27b2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:57:03.074657 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:03.074620    4891 patch_prober.go:29] interesting pod/image-registry-dcff7b7b8-bck5q container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 14:57:03.075130 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:03.074686    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q" podUID="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:57:03.683879 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:03.683850    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f6fgv"
Nov 02 14:57:05.775140 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:05.775107    4891 patch_prober.go:29] interesting pod/image-registry-dcff7b7b8-bck5q container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 14:57:05.775547 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:05.775157    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q" podUID="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:57:06.135941 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.135860    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v2p4h"]
Nov 02 14:57:06.170356 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.170307    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.272269 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.272233    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 14:57:06.272469 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.272417    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 14:57:06.272469 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.272432    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 14:57:06.272469 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.272430    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 14:57:06.272568 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.272483    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zx8m7\""
Nov 02 14:57:06.290019 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.289988    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/170d86eb-a48d-4125-b264-fff55074f210-root\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.290091 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.290025    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/170d86eb-a48d-4125-b264-fff55074f210-sys\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.290091 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.290065    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkjsc\" (UniqueName: \"kubernetes.io/projected/170d86eb-a48d-4125-b264-fff55074f210-kube-api-access-rkjsc\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.290153 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.290147    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-wtmp\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.290185 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.290171    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.290225 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.290197    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/170d86eb-a48d-4125-b264-fff55074f210-metrics-client-ca\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.290225 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.290218    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-textfile\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.290302 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.290236    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-tls\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.290423 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.290400    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.390859 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.390778    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.390859 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.390822    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/170d86eb-a48d-4125-b264-fff55074f210-root\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.390859 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.390853    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/170d86eb-a48d-4125-b264-fff55074f210-sys\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.391092 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.390882    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkjsc\" (UniqueName: \"kubernetes.io/projected/170d86eb-a48d-4125-b264-fff55074f210-kube-api-access-rkjsc\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.391092 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.390914    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-wtmp\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.391092 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.390951    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.391092 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.390962    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/170d86eb-a48d-4125-b264-fff55074f210-root\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.391092 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.390975    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/170d86eb-a48d-4125-b264-fff55074f210-sys\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.391092 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.390995    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/170d86eb-a48d-4125-b264-fff55074f210-metrics-client-ca\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.391092 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.391044    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-textfile\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.391092 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.391069    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-tls\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.391092 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.391082    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-wtmp\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.391549 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.391422    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-textfile\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.391594 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.391548    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.391594 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.391579    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/170d86eb-a48d-4125-b264-fff55074f210-metrics-client-ca\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.393140 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.393112    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.393370 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.393352    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/170d86eb-a48d-4125-b264-fff55074f210-node-exporter-tls\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.452631 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.452601    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkjsc\" (UniqueName: \"kubernetes.io/projected/170d86eb-a48d-4125-b264-fff55074f210-kube-api-access-rkjsc\") pod \"node-exporter-v2p4h\" (UID: \"170d86eb-a48d-4125-b264-fff55074f210\") " pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.516771 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.516737    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2p4h"
Nov 02 14:57:06.556129 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:57:06.556097    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170d86eb_a48d_4125_b264_fff55074f210.slice/crio-25cf170a39f7ac46b1e87dfd9fad3ba2e198b7c91ae6f33713bd34da08f62d12 WatchSource:0}: Error finding container 25cf170a39f7ac46b1e87dfd9fad3ba2e198b7c91ae6f33713bd34da08f62d12: Status 404 returned error can't find the container with id 25cf170a39f7ac46b1e87dfd9fad3ba2e198b7c91ae6f33713bd34da08f62d12
Nov 02 14:57:06.724953 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.724865    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2p4h" event={"ID":"170d86eb-a48d-4125-b264-fff55074f210","Type":"ContainerStarted","Data":"25cf170a39f7ac46b1e87dfd9fad3ba2e198b7c91ae6f33713bd34da08f62d12"}
Nov 02 14:57:06.921235 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:06.921202    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jjhkj"
Nov 02 14:57:08.735084 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:08.734996    4891 generic.go:356] "Generic (PLEG): container finished" podID="170d86eb-a48d-4125-b264-fff55074f210" containerID="c93523a23bfc5f9ca987ed4bc643091662067de45f9b890e3f109825aef0e395" exitCode=0
Nov 02 14:57:08.735509 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:08.735083    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2p4h" event={"ID":"170d86eb-a48d-4125-b264-fff55074f210","Type":"ContainerDied","Data":"c93523a23bfc5f9ca987ed4bc643091662067de45f9b890e3f109825aef0e395"}
Nov 02 14:57:09.741156 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:09.741118    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2p4h" event={"ID":"170d86eb-a48d-4125-b264-fff55074f210","Type":"ContainerStarted","Data":"61e790b314b3c747b43d0fd181f0cd0cde57c1d323314a683c1bf8bb5266b92f"}
Nov 02 14:57:09.741560 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:09.741171    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2p4h" event={"ID":"170d86eb-a48d-4125-b264-fff55074f210","Type":"ContainerStarted","Data":"ce1a34b4fb3e9b2ded162aac59963d4c7857d0b444b410e933c5b0cbf50c247a"}
Nov 02 14:57:09.860853 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:09.860798    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v2p4h" podStartSLOduration=1.873833539 podStartE2EDuration="3.860782598s" totalImagesPullingTime="1.916617367s" totalInitContainerRuntime="70.331692ms" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:09.859778685 +0000 UTC m=+37.452395484" watchObservedRunningTime="2026-11-02 14:57:09.860782598 +0000 UTC m=+37.453399396"
Nov 02 14:57:11.933301 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:11.933260    4891 patch_prober.go:29] interesting pod/image-registry-74fd4fbb69-thwc5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 14:57:11.933700 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:11.933329    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5" podUID="8d2ae30e-0191-41ff-85da-dedf17ad27b2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:57:14.273357 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:57:14.273265    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 14:57:14.574945 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:14.574861    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dcff7b7b8-bck5q"]
Nov 02 14:57:14.578948 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:14.578920    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:57:16.959576 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:16.959512    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5" podUID="8d2ae30e-0191-41ff-85da-dedf17ad27b2" containerName="registry" containerID="cri-o://b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4" gracePeriod=30
Nov 02 14:57:17.194419 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.194395    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:57:17.276740 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.276711    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-certificates\") pod \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") "
Nov 02 14:57:17.276902 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.276758    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-image-registry-private-configuration\") pod \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") "
Nov 02 14:57:17.276902 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.276783    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-trusted-ca\") pod \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") "
Nov 02 14:57:17.276902 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.276819    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") pod \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") "
Nov 02 14:57:17.277052 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.277028    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-kube-api-access-2fwhf\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-kube-api-access-2fwhf\") pod \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") "
Nov 02 14:57:17.277109 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.277093    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-bound-sa-token\") pod \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") "
Nov 02 14:57:17.277166 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.277133    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-installation-pull-secrets\") pod \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") "
Nov 02 14:57:17.277219 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.277174    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-certificates" pod "8d2ae30e-0191-41ff-85da-dedf17ad27b2" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:17.277219 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.277187    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2ae30e-0191-41ff-85da-dedf17ad27b2-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d2ae30e-0191-41ff-85da-dedf17ad27b2-ca-trust-extracted\") pod \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\" (UID: \"8d2ae30e-0191-41ff-85da-dedf17ad27b2\") "
Nov 02 14:57:17.277352 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.277210    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-trusted-ca" pod "8d2ae30e-0191-41ff-85da-dedf17ad27b2" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:17.277470 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.277442    4891 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-certificates\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:17.277533 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.277470    4891 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d2ae30e-0191-41ff-85da-dedf17ad27b2-trusted-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:17.279413 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.279365    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-image-registry-private-configuration" pod "8d2ae30e-0191-41ff-85da-dedf17ad27b2" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:17.279557 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.279451    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls" pod "8d2ae30e-0191-41ff-85da-dedf17ad27b2" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:17.279557 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.279506    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-installation-pull-secrets" pod "8d2ae30e-0191-41ff-85da-dedf17ad27b2" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:17.279637 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.279595    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-bound-sa-token" pod "8d2ae30e-0191-41ff-85da-dedf17ad27b2" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:17.279887 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.279858    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-kube-api-access-2fwhf" pod "8d2ae30e-0191-41ff-85da-dedf17ad27b2" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2"). InnerVolumeSpecName "kube-api-access-2fwhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:17.284297 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.284274    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2ae30e-0191-41ff-85da-dedf17ad27b2-ca-trust-extracted" pod "8d2ae30e-0191-41ff-85da-dedf17ad27b2" (UID: "8d2ae30e-0191-41ff-85da-dedf17ad27b2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.378493 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.378461    4891 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d2ae30e-0191-41ff-85da-dedf17ad27b2-ca-trust-extracted\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:17.378493 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.378493    4891 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-image-registry-private-configuration\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:17.378689 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.378510    4891 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-registry-tls\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:17.378689 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.378523    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fwhf\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-kube-api-access-2fwhf\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:17.378689 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.378537    4891 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d2ae30e-0191-41ff-85da-dedf17ad27b2-bound-sa-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:17.378689 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.378549    4891 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d2ae30e-0191-41ff-85da-dedf17ad27b2-installation-pull-secrets\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:17.772002 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.771967    4891 generic.go:356] "Generic (PLEG): container finished" podID="8d2ae30e-0191-41ff-85da-dedf17ad27b2" containerID="b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4" exitCode=0
Nov 02 14:57:17.772274 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.772016    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5"
Nov 02 14:57:17.772274 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.772058    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5" event={"ID":"8d2ae30e-0191-41ff-85da-dedf17ad27b2","Type":"ContainerDied","Data":"b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4"}
Nov 02 14:57:17.772274 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.772114    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74fd4fbb69-thwc5" event={"ID":"8d2ae30e-0191-41ff-85da-dedf17ad27b2","Type":"ContainerDied","Data":"33da889bfc3403307eb6f2afeb4fffeca757b1f4c60fee8c7737afa3ef247f74"}
Nov 02 14:57:17.772274 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.772134    4891 scope.go:122] "RemoveContainer" containerID="b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4"
Nov 02 14:57:17.780581 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.780564    4891 scope.go:122] "RemoveContainer" containerID="b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4"
Nov 02 14:57:17.780827 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:57:17.780809    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4\": container with ID starting with b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4 not found: ID does not exist" containerID="b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4"
Nov 02 14:57:17.780868 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.780838    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4"} err="failed to get container status \"b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4\": rpc error: code = NotFound desc = could not find container \"b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4\": container with ID starting with b3446cb9a5fc95ee85d550cb9ff10c19e15856c4fa39b077686618aa970792b4 not found: ID does not exist"
Nov 02 14:57:17.935702 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.935665    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74fd4fbb69-thwc5"]
Nov 02 14:57:17.985350 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:17.985305    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74fd4fbb69-thwc5"]
Nov 02 14:57:18.519640 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:18.519601    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2ae30e-0191-41ff-85da-dedf17ad27b2" path="/var/lib/kubelet/pods/8d2ae30e-0191-41ff-85da-dedf17ad27b2/volumes"
Nov 02 14:57:38.848164 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:38.848130    4891 generic.go:356] "Generic (PLEG): container finished" podID="063ba34d-7544-410f-b861-5c300cc8f6d4" containerID="1c2a61e5e15f57f689fafa9930029d09bc8798cf59e550b9151a9a4b40c497c5" exitCode=0
Nov 02 14:57:38.848601 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:38.848204    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ggtvw" event={"ID":"063ba34d-7544-410f-b861-5c300cc8f6d4","Type":"ContainerDied","Data":"1c2a61e5e15f57f689fafa9930029d09bc8798cf59e550b9151a9a4b40c497c5"}
Nov 02 14:57:38.848664 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:38.848598    4891 scope.go:122] "RemoveContainer" containerID="1c2a61e5e15f57f689fafa9930029d09bc8798cf59e550b9151a9a4b40c497c5"
Nov 02 14:57:38.849845 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:38.849822    4891 generic.go:356] "Generic (PLEG): container finished" podID="234a4a99-5491-44ef-b58c-3e4ab32584e5" containerID="82f34400af35b3b9480695ff79e0ec70c83d42a7843156ed1c2ccf25cfcb6e6d" exitCode=0
Nov 02 14:57:38.849946 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:38.849904    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d" event={"ID":"234a4a99-5491-44ef-b58c-3e4ab32584e5","Type":"ContainerDied","Data":"82f34400af35b3b9480695ff79e0ec70c83d42a7843156ed1c2ccf25cfcb6e6d"}
Nov 02 14:57:38.850152 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:38.850139    4891 scope.go:122] "RemoveContainer" containerID="82f34400af35b3b9480695ff79e0ec70c83d42a7843156ed1c2ccf25cfcb6e6d"
Nov 02 14:57:39.594465 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.594417    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q" podUID="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" containerName="registry" containerID="cri-o://1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321" gracePeriod=30
Nov 02 14:57:39.847975 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.847924    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:57:39.855274 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.855246    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lz25d" event={"ID":"234a4a99-5491-44ef-b58c-3e4ab32584e5","Type":"ContainerStarted","Data":"b2751aa43a7ce12c9a938e54d48e85b720a138ac1862390e7fd891132ae326b3"}
Nov 02 14:57:39.856428 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.856406    4891 generic.go:356] "Generic (PLEG): container finished" podID="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" containerID="1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321" exitCode=0
Nov 02 14:57:39.856532 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.856436    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q"
Nov 02 14:57:39.856532 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.856459    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q" event={"ID":"fac285e0-f60a-4f53-a7e8-dee42bb83a0a","Type":"ContainerDied","Data":"1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321"}
Nov 02 14:57:39.856532 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.856510    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dcff7b7b8-bck5q" event={"ID":"fac285e0-f60a-4f53-a7e8-dee42bb83a0a","Type":"ContainerDied","Data":"d67cfbb786aa3abb4518fabf0aac1db65d43a53c8784a03c7524930433188639"}
Nov 02 14:57:39.856690 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.856537    4891 scope.go:122] "RemoveContainer" containerID="1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321"
Nov 02 14:57:39.858253 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.858233    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-ggtvw" event={"ID":"063ba34d-7544-410f-b861-5c300cc8f6d4","Type":"ContainerStarted","Data":"5d2bcf57037c5205978b418487ac362caad86aba198b01b4c0543c9233c6a80a"}
Nov 02 14:57:39.865240 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.865218    4891 scope.go:122] "RemoveContainer" containerID="1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321"
Nov 02 14:57:39.865591 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:57:39.865571    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321\": container with ID starting with 1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321 not found: ID does not exist" containerID="1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321"
Nov 02 14:57:39.865692 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.865599    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321"} err="failed to get container status \"1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321\": rpc error: code = NotFound desc = could not find container \"1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321\": container with ID starting with 1cca50bd6369f3a2fe0f2358ef5871ff11dba8aa6d1185221ac1231511e68321 not found: ID does not exist"
Nov 02 14:57:39.960856 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.960822    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-bound-sa-token\") pod \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") "
Nov 02 14:57:39.961046 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.960889    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-ca-trust-extracted\") pod \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") "
Nov 02 14:57:39.961046 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.960923    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-installation-pull-secrets\") pod \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") "
Nov 02 14:57:39.961046 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.960947    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-kube-api-access-qbskj\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-kube-api-access-qbskj\") pod \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") "
Nov 02 14:57:39.961046 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.960972    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-trusted-ca\") pod \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") "
Nov 02 14:57:39.961269 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.961119    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-certificates\") pod \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") "
Nov 02 14:57:39.961269 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.961173    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-image-registry-private-configuration\") pod \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") "
Nov 02 14:57:39.961269 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.961221    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") pod \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\" (UID: \"fac285e0-f60a-4f53-a7e8-dee42bb83a0a\") "
Nov 02 14:57:39.961614 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.961570    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-certificates" pod "fac285e0-f60a-4f53-a7e8-dee42bb83a0a" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:39.962050 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.962023    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-trusted-ca" pod "fac285e0-f60a-4f53-a7e8-dee42bb83a0a" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:39.963788 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.963743    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-installation-pull-secrets" pod "fac285e0-f60a-4f53-a7e8-dee42bb83a0a" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:39.963944 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.963925    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-bound-sa-token" pod "fac285e0-f60a-4f53-a7e8-dee42bb83a0a" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:39.964060 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.964037    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-kube-api-access-qbskj" pod "fac285e0-f60a-4f53-a7e8-dee42bb83a0a" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a"). InnerVolumeSpecName "kube-api-access-qbskj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:39.964098 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.964076    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-image-registry-private-configuration" pod "fac285e0-f60a-4f53-a7e8-dee42bb83a0a" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:39.964731 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.964708    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls" pod "fac285e0-f60a-4f53-a7e8-dee42bb83a0a" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:39.969116 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:39.969091    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-ca-trust-extracted" pod "fac285e0-f60a-4f53-a7e8-dee42bb83a0a" (UID: "fac285e0-f60a-4f53-a7e8-dee42bb83a0a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:40.062525 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.062491    4891 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-bound-sa-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:40.062525 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.062518    4891 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-ca-trust-extracted\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:40.062525 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.062527    4891 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-installation-pull-secrets\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:40.062758 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.062536    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbskj\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-kube-api-access-qbskj\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:40.062758 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.062546    4891 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-trusted-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:40.062758 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.062554    4891 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-certificates\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:40.062758 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.062564    4891 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-image-registry-private-configuration\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:40.062758 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.062574    4891 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fac285e0-f60a-4f53-a7e8-dee42bb83a0a-registry-tls\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:57:40.292154 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.292112    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-7xfml-d96n7"]
Nov 02 14:57:40.292497 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.292480    4891 cpu_manager.go:414] "containerMap: removing container" podUID="8d2ae30e-0191-41ff-85da-dedf17ad27b2" containerName="registry"
Nov 02 14:57:40.292581 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.292499    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2ae30e-0191-41ff-85da-dedf17ad27b2" containerName="registry"
Nov 02 14:57:40.292581 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.292516    4891 cpu_manager.go:414] "containerMap: removing container" podUID="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" containerName="registry"
Nov 02 14:57:40.292581 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.292525    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" containerName="registry"
Nov 02 14:57:40.292728 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.292640    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2ae30e-0191-41ff-85da-dedf17ad27b2" containerName="registry"
Nov 02 14:57:40.292728 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.292653    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" containerName="registry"
Nov 02 14:57:40.295598 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.295576    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.354684 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.354651    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7xfml-d96n7"]
Nov 02 14:57:40.363593 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.363562    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-trusted-ca-bundle\") pod \"periodic-gathering-7xfml-d96n7\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") " pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.363759 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.363613    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-service-ca-bundle\") pod \"periodic-gathering-7xfml-d96n7\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") " pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.363759 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.363685    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6ad99afd-47d8-437e-9898-3fe9d10f63b9-archives-path\") pod \"periodic-gathering-7xfml-d96n7\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") " pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.363895 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.363773    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv4wb\" (UniqueName: \"kubernetes.io/projected/6ad99afd-47d8-437e-9898-3fe9d10f63b9-kube-api-access-wv4wb\") pod \"periodic-gathering-7xfml-d96n7\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") " pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.417470 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.417391    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dcff7b7b8-bck5q"]
Nov 02 14:57:40.421516 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.421489    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-dcff7b7b8-bck5q"]
Nov 02 14:57:40.464111 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.464079    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-trusted-ca-bundle\") pod \"periodic-gathering-7xfml-d96n7\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") " pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.464267 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.464134    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-service-ca-bundle\") pod \"periodic-gathering-7xfml-d96n7\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") " pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.464267 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.464179    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6ad99afd-47d8-437e-9898-3fe9d10f63b9-archives-path\") pod \"periodic-gathering-7xfml-d96n7\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") " pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.464267 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.464213    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4wb\" (UniqueName: \"kubernetes.io/projected/6ad99afd-47d8-437e-9898-3fe9d10f63b9-kube-api-access-wv4wb\") pod \"periodic-gathering-7xfml-d96n7\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") " pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.464637 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.464615    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6ad99afd-47d8-437e-9898-3fe9d10f63b9-archives-path\") pod \"periodic-gathering-7xfml-d96n7\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") " pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.464839 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.464819    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-service-ca-bundle\") pod \"periodic-gathering-7xfml-d96n7\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") " pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.465213 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.465196    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-trusted-ca-bundle\") pod \"periodic-gathering-7xfml-d96n7\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") " pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.519565 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.519533    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac285e0-f60a-4f53-a7e8-dee42bb83a0a" path="/var/lib/kubelet/pods/fac285e0-f60a-4f53-a7e8-dee42bb83a0a/volumes"
Nov 02 14:57:40.525433 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.525412    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4wb\" (UniqueName: \"kubernetes.io/projected/6ad99afd-47d8-437e-9898-3fe9d10f63b9-kube-api-access-wv4wb\") pod \"periodic-gathering-7xfml-d96n7\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") " pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.605685 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.605648    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:57:40.977638 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:40.977604    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7xfml-d96n7"]
Nov 02 14:57:40.981406 ip-10-0-132-27 kubenswrapper[4891]: W1102 14:57:40.981378    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad99afd_47d8_437e_9898_3fe9d10f63b9.slice/crio-82b92232253164f01a38f9657323c8299a5cd353b63f6f1b21e6ee2d7a4d0e2e WatchSource:0}: Error finding container 82b92232253164f01a38f9657323c8299a5cd353b63f6f1b21e6ee2d7a4d0e2e: Status 404 returned error can't find the container with id 82b92232253164f01a38f9657323c8299a5cd353b63f6f1b21e6ee2d7a4d0e2e
Nov 02 14:57:41.867914 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:41.867876    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7xfml-d96n7" event={"ID":"6ad99afd-47d8-437e-9898-3fe9d10f63b9","Type":"ContainerStarted","Data":"1485ac20ce87e5dac54bdc9f995df42c752ef9ed00f8b23591f0d5f104920a73"}
Nov 02 14:57:41.868098 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:41.867961    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7xfml-d96n7" event={"ID":"6ad99afd-47d8-437e-9898-3fe9d10f63b9","Type":"ContainerStarted","Data":"82b92232253164f01a38f9657323c8299a5cd353b63f6f1b21e6ee2d7a4d0e2e"}
Nov 02 14:57:41.984125 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:57:41.984075    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-7xfml-d96n7" podStartSLOduration=1.984061659 podStartE2EDuration="1.984061659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:41.983779308 +0000 UTC m=+69.576396102" watchObservedRunningTime="2026-11-02 14:57:41.984061659 +0000 UTC m=+69.576678457"
Nov 02 14:57:44.273059 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:57:44.273021    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 14:58:04.949474 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:58:04.949440    4891 generic.go:356] "Generic (PLEG): container finished" podID="fc12a767-3259-4dc0-a809-d123b58334e3" containerID="7afdbaf07390958b000f5acf887818ad28facc2349d602026d8d605d0bca32df" exitCode=0
Nov 02 14:58:04.949898 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:58:04.949480    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb" event={"ID":"fc12a767-3259-4dc0-a809-d123b58334e3","Type":"ContainerDied","Data":"7afdbaf07390958b000f5acf887818ad28facc2349d602026d8d605d0bca32df"}
Nov 02 14:58:04.949898 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:58:04.949782    4891 scope.go:122] "RemoveContainer" containerID="7afdbaf07390958b000f5acf887818ad28facc2349d602026d8d605d0bca32df"
Nov 02 14:58:05.956194 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:58:05.956160    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wbwdb" event={"ID":"fc12a767-3259-4dc0-a809-d123b58334e3","Type":"ContainerStarted","Data":"4751700312d670f557fbe19d95a659af85870b07681f2149f44265d7fb8fc72b"}
Nov 02 14:58:14.273228 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:58:14.273193    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 14:58:44.272662 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:58:44.272567    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 14:59:07.162165 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:07.162131    4891 generic.go:356] "Generic (PLEG): container finished" podID="6ad99afd-47d8-437e-9898-3fe9d10f63b9" containerID="1485ac20ce87e5dac54bdc9f995df42c752ef9ed00f8b23591f0d5f104920a73" exitCode=0
Nov 02 14:59:07.162535 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:07.162206    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7xfml-d96n7" event={"ID":"6ad99afd-47d8-437e-9898-3fe9d10f63b9","Type":"ContainerDied","Data":"1485ac20ce87e5dac54bdc9f995df42c752ef9ed00f8b23591f0d5f104920a73"}
Nov 02 14:59:08.310212 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.310185    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:59:08.427458 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.427379    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-trusted-ca-bundle\") pod \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") "
Nov 02 14:59:08.427458 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.427452    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad99afd-47d8-437e-9898-3fe9d10f63b9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6ad99afd-47d8-437e-9898-3fe9d10f63b9-archives-path\") pod \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") "
Nov 02 14:59:08.427700 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.427484    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad99afd-47d8-437e-9898-3fe9d10f63b9-kube-api-access-wv4wb\" (UniqueName: \"kubernetes.io/projected/6ad99afd-47d8-437e-9898-3fe9d10f63b9-kube-api-access-wv4wb\") pod \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") "
Nov 02 14:59:08.427700 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.427517    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-service-ca-bundle\") pod \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\" (UID: \"6ad99afd-47d8-437e-9898-3fe9d10f63b9\") "
Nov 02 14:59:08.427834 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.427805    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-trusted-ca-bundle" pod "6ad99afd-47d8-437e-9898-3fe9d10f63b9" (UID: "6ad99afd-47d8-437e-9898-3fe9d10f63b9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:59:08.427988 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.427884    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad99afd-47d8-437e-9898-3fe9d10f63b9-archives-path" pod "6ad99afd-47d8-437e-9898-3fe9d10f63b9" (UID: "6ad99afd-47d8-437e-9898-3fe9d10f63b9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:59:08.428069 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.428013    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-service-ca-bundle" pod "6ad99afd-47d8-437e-9898-3fe9d10f63b9" (UID: "6ad99afd-47d8-437e-9898-3fe9d10f63b9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:59:08.429899 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.429874    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad99afd-47d8-437e-9898-3fe9d10f63b9-kube-api-access-wv4wb" pod "6ad99afd-47d8-437e-9898-3fe9d10f63b9" (UID: "6ad99afd-47d8-437e-9898-3fe9d10f63b9"). InnerVolumeSpecName "kube-api-access-wv4wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:59:08.528432 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.528397    4891 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-trusted-ca-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:59:08.528432 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.528431    4891 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6ad99afd-47d8-437e-9898-3fe9d10f63b9-archives-path\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:59:08.528616 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.528444    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv4wb\" (UniqueName: \"kubernetes.io/projected/6ad99afd-47d8-437e-9898-3fe9d10f63b9-kube-api-access-wv4wb\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:59:08.528616 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:08.528454    4891 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad99afd-47d8-437e-9898-3fe9d10f63b9-service-ca-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:59:09.170369 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:09.170327    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7xfml-d96n7" event={"ID":"6ad99afd-47d8-437e-9898-3fe9d10f63b9","Type":"ContainerDied","Data":"82b92232253164f01a38f9657323c8299a5cd353b63f6f1b21e6ee2d7a4d0e2e"}
Nov 02 14:59:09.170369 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:09.170350    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7xfml-d96n7"
Nov 02 14:59:09.170583 ip-10-0-132-27 kubenswrapper[4891]: I1102 14:59:09.170409    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82b92232253164f01a38f9657323c8299a5cd353b63f6f1b21e6ee2d7a4d0e2e"
Nov 02 14:59:14.272479 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:59:14.272435    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 14:59:44.272400 ip-10-0-132-27 kubenswrapper[4891]: E1102 14:59:44.272362    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:00:07.722647 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.722609    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"]
Nov 02 15:00:07.723254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.723058    4891 cpu_manager.go:414] "containerMap: removing container" podUID="6ad99afd-47d8-437e-9898-3fe9d10f63b9" containerName="insights-gathering"
Nov 02 15:00:07.723254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.723078    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad99afd-47d8-437e-9898-3fe9d10f63b9" containerName="insights-gathering"
Nov 02 15:00:07.723254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.723168    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad99afd-47d8-437e-9898-3fe9d10f63b9" containerName="insights-gathering"
Nov 02 15:00:07.726254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.726239    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:07.828075 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.828036    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2gsxf\""
Nov 02 15:00:07.828075 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.828062    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 15:00:07.828362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.828192    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 15:00:07.829138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.829118    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"]
Nov 02 15:00:07.829812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.829791    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"]
Nov 02 15:00:07.833034 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.833019    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:07.880438 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.880400    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2\" (UID: \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:07.880608 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.880449    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2\" (UID: \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:07.880608 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.880509    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tdkg\" (UniqueName: \"kubernetes.io/projected/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-kube-api-access-7tdkg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2\" (UID: \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:07.906672 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.906640    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7rb6r\""
Nov 02 15:00:07.928370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.928343    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"]
Nov 02 15:00:07.981182 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.981100    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f\" (UID: \"1f934255-4e4b-4d05-91a7-15ce08f174e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:07.981182 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.981153    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2\" (UID: \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:07.981182 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.981173    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ht8\" (UniqueName: \"kubernetes.io/projected/1f934255-4e4b-4d05-91a7-15ce08f174e4-kube-api-access-v8ht8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f\" (UID: \"1f934255-4e4b-4d05-91a7-15ce08f174e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:07.981416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.981226    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f\" (UID: \"1f934255-4e4b-4d05-91a7-15ce08f174e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:07.981454 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.981410    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2\" (UID: \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:07.981454 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.981442    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tdkg\" (UniqueName: \"kubernetes.io/projected/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-kube-api-access-7tdkg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2\" (UID: \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:07.981573 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.981554    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2\" (UID: \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:07.981718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:07.981703    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2\" (UID: \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:08.044912 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:08.044876    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tdkg\" (UniqueName: \"kubernetes.io/projected/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-kube-api-access-7tdkg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2\" (UID: \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:08.081996 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:08.081963    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ht8\" (UniqueName: \"kubernetes.io/projected/1f934255-4e4b-4d05-91a7-15ce08f174e4-kube-api-access-v8ht8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f\" (UID: \"1f934255-4e4b-4d05-91a7-15ce08f174e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:08.081996 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:08.082002    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f\" (UID: \"1f934255-4e4b-4d05-91a7-15ce08f174e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:08.082221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:08.082051    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f\" (UID: \"1f934255-4e4b-4d05-91a7-15ce08f174e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:08.082472 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:08.082455    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f\" (UID: \"1f934255-4e4b-4d05-91a7-15ce08f174e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:08.082523 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:08.082486    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f\" (UID: \"1f934255-4e4b-4d05-91a7-15ce08f174e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:08.143165 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:08.143127    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ht8\" (UniqueName: \"kubernetes.io/projected/1f934255-4e4b-4d05-91a7-15ce08f174e4-kube-api-access-v8ht8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f\" (UID: \"1f934255-4e4b-4d05-91a7-15ce08f174e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:08.148929 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:08.148908    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:08.341633 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:08.341599    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:08.533823 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:08.533791    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"]
Nov 02 15:00:08.536997 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:08.536975    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f934255_4e4b_4d05_91a7_15ce08f174e4.slice/crio-b811ea97262b73966d726cb893a09382c806eadc2b98528190f1dea0555b6b6c WatchSource:0}: Error finding container b811ea97262b73966d726cb893a09382c806eadc2b98528190f1dea0555b6b6c: Status 404 returned error can't find the container with id b811ea97262b73966d726cb893a09382c806eadc2b98528190f1dea0555b6b6c
Nov 02 15:00:08.714543 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:08.714471    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"]
Nov 02 15:00:08.717334 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:08.717281    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6655aa2e_f88a_45f2_a5d7_927a9802ab4c.slice/crio-e1179b03cbf2261bb34b66c018aedf85fcdb60857b42bf9fb4c3cab3ff826da8 WatchSource:0}: Error finding container e1179b03cbf2261bb34b66c018aedf85fcdb60857b42bf9fb4c3cab3ff826da8: Status 404 returned error can't find the container with id e1179b03cbf2261bb34b66c018aedf85fcdb60857b42bf9fb4c3cab3ff826da8
Nov 02 15:00:09.373898 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:09.373861    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f" event={"ID":"1f934255-4e4b-4d05-91a7-15ce08f174e4","Type":"ContainerStarted","Data":"b811ea97262b73966d726cb893a09382c806eadc2b98528190f1dea0555b6b6c"}
Nov 02 15:00:09.374987 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:09.374964    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2" event={"ID":"6655aa2e-f88a-45f2-a5d7-927a9802ab4c","Type":"ContainerStarted","Data":"e1179b03cbf2261bb34b66c018aedf85fcdb60857b42bf9fb4c3cab3ff826da8"}
Nov 02 15:00:10.193713 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.193679    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"]
Nov 02 15:00:10.197603 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.197579    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:10.299592 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.299536    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-hz57l\""
Nov 02 15:00:10.300831 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.300805    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"]
Nov 02 15:00:10.301072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.301042    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x\" (UID: \"f3b70660-5510-4dd2-9e89-83cc2193fc93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:10.301178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.301090    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7hjr\" (UniqueName: \"kubernetes.io/projected/f3b70660-5510-4dd2-9e89-83cc2193fc93-kube-api-access-x7hjr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x\" (UID: \"f3b70660-5510-4dd2-9e89-83cc2193fc93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:10.301247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.301232    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x\" (UID: \"f3b70660-5510-4dd2-9e89-83cc2193fc93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:10.402380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.402347    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x\" (UID: \"f3b70660-5510-4dd2-9e89-83cc2193fc93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:10.402380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.402384    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7hjr\" (UniqueName: \"kubernetes.io/projected/f3b70660-5510-4dd2-9e89-83cc2193fc93-kube-api-access-x7hjr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x\" (UID: \"f3b70660-5510-4dd2-9e89-83cc2193fc93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:10.402982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.402418    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x\" (UID: \"f3b70660-5510-4dd2-9e89-83cc2193fc93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:10.402982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.402818    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x\" (UID: \"f3b70660-5510-4dd2-9e89-83cc2193fc93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:10.403136 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.403047    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x\" (UID: \"f3b70660-5510-4dd2-9e89-83cc2193fc93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:10.465193 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.465123    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7hjr\" (UniqueName: \"kubernetes.io/projected/f3b70660-5510-4dd2-9e89-83cc2193fc93-kube-api-access-x7hjr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x\" (UID: \"f3b70660-5510-4dd2-9e89-83cc2193fc93\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:10.517882 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.517835    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:10.903330 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:10.903219    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"]
Nov 02 15:00:10.907437 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:10.907402    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b70660_5510_4dd2_9e89_83cc2193fc93.slice/crio-5d8c16a3d1bb1f0fce288f7938d6bce25c9de16e964960b839be1dbe68ea95d0 WatchSource:0}: Error finding container 5d8c16a3d1bb1f0fce288f7938d6bce25c9de16e964960b839be1dbe68ea95d0: Status 404 returned error can't find the container with id 5d8c16a3d1bb1f0fce288f7938d6bce25c9de16e964960b839be1dbe68ea95d0
Nov 02 15:00:11.386778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:11.386738    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x" event={"ID":"f3b70660-5510-4dd2-9e89-83cc2193fc93","Type":"ContainerStarted","Data":"5d8c16a3d1bb1f0fce288f7938d6bce25c9de16e964960b839be1dbe68ea95d0"}
Nov 02 15:00:13.093884 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.093798    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"]
Nov 02 15:00:13.098915 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.098885    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:13.129105 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.128928    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92pbh\" (UniqueName: \"kubernetes.io/projected/1c5c0540-2328-49c6-b198-57d221d88c0a-kube-api-access-92pbh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22\" (UID: \"1c5c0540-2328-49c6-b198-57d221d88c0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:13.129105 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.128974    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22\" (UID: \"1c5c0540-2328-49c6-b198-57d221d88c0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:13.129105 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.129016    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22\" (UID: \"1c5c0540-2328-49c6-b198-57d221d88c0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:13.204240 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.204187    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-2d6m2\""
Nov 02 15:00:13.207402 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.207358    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"]
Nov 02 15:00:13.207559 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.207435    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"]
Nov 02 15:00:13.211397 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.211374    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:13.229801 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.229770    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22\" (UID: \"1c5c0540-2328-49c6-b198-57d221d88c0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:13.229939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.229836    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92pbh\" (UniqueName: \"kubernetes.io/projected/1c5c0540-2328-49c6-b198-57d221d88c0a-kube-api-access-92pbh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22\" (UID: \"1c5c0540-2328-49c6-b198-57d221d88c0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:13.229939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.229873    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd8w7\" (UniqueName: \"kubernetes.io/projected/c0b8c1b2-8f94-40f0-a758-f1027a797b69-kube-api-access-zd8w7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2\" (UID: \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:13.229939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.229906    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22\" (UID: \"1c5c0540-2328-49c6-b198-57d221d88c0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:13.230115 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.229943    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2\" (UID: \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:13.230115 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.229966    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2\" (UID: \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:13.230224 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.230188    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22\" (UID: \"1c5c0540-2328-49c6-b198-57d221d88c0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:13.230423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.230397    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22\" (UID: \"1c5c0540-2328-49c6-b198-57d221d88c0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:13.269415 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.269384    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-tnp47\""
Nov 02 15:00:13.270742 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.270701    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"]
Nov 02 15:00:13.291585 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.291548    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"]
Nov 02 15:00:13.294081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.294048    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92pbh\" (UniqueName: \"kubernetes.io/projected/1c5c0540-2328-49c6-b198-57d221d88c0a-kube-api-access-92pbh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22\" (UID: \"1c5c0540-2328-49c6-b198-57d221d88c0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:13.296248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.295805    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:13.331532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.331095    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd8w7\" (UniqueName: \"kubernetes.io/projected/c0b8c1b2-8f94-40f0-a758-f1027a797b69-kube-api-access-zd8w7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2\" (UID: \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:13.331532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.331164    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2\" (UID: \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:13.331532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.331193    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2\" (UID: \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:13.331532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.331253    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr\" (UID: \"e1624891-a22f-4ed4-8590-f44983e1fbbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:13.331532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.331384    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sskkd\" (UniqueName: \"kubernetes.io/projected/e1624891-a22f-4ed4-8590-f44983e1fbbb-kube-api-access-sskkd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr\" (UID: \"e1624891-a22f-4ed4-8590-f44983e1fbbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:13.331532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.331420    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr\" (UID: \"e1624891-a22f-4ed4-8590-f44983e1fbbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:13.331995 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.331616    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2\" (UID: \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:13.331995 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.331746    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2\" (UID: \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:13.351992 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.351908    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"]
Nov 02 15:00:13.367744 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.367706    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-dttsm\""
Nov 02 15:00:13.394709 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.394678    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"]
Nov 02 15:00:13.396187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.396159    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd8w7\" (UniqueName: \"kubernetes.io/projected/c0b8c1b2-8f94-40f0-a758-f1027a797b69-kube-api-access-zd8w7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2\" (UID: \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:13.403019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.402802    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:13.419734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.419704    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:13.432854 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.432809    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjcbb\" (UniqueName: \"kubernetes.io/projected/b0086234-e43e-40f0-be10-217afb3ba74d-kube-api-access-wjcbb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m\" (UID: \"b0086234-e43e-40f0-be10-217afb3ba74d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:13.432966 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.432895    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr\" (UID: \"e1624891-a22f-4ed4-8590-f44983e1fbbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:13.432966 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.432936    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m\" (UID: \"b0086234-e43e-40f0-be10-217afb3ba74d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:13.433088 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.432975    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sskkd\" (UniqueName: \"kubernetes.io/projected/e1624891-a22f-4ed4-8590-f44983e1fbbb-kube-api-access-sskkd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr\" (UID: \"e1624891-a22f-4ed4-8590-f44983e1fbbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:13.433088 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.433005    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr\" (UID: \"e1624891-a22f-4ed4-8590-f44983e1fbbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:13.433088 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.433052    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m\" (UID: \"b0086234-e43e-40f0-be10-217afb3ba74d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:13.433524 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.433501    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr\" (UID: \"e1624891-a22f-4ed4-8590-f44983e1fbbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:13.433803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.433780    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr\" (UID: \"e1624891-a22f-4ed4-8590-f44983e1fbbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:13.457434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.457397    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"]
Nov 02 15:00:13.470446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.468708    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-m5x99\""
Nov 02 15:00:13.504524 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.504493    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sskkd\" (UniqueName: \"kubernetes.io/projected/e1624891-a22f-4ed4-8590-f44983e1fbbb-kube-api-access-sskkd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr\" (UID: \"e1624891-a22f-4ed4-8590-f44983e1fbbb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:13.532060 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.532030    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:13.535637 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.534408    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m\" (UID: \"b0086234-e43e-40f0-be10-217afb3ba74d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:13.535637 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.534474    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m\" (UID: \"b0086234-e43e-40f0-be10-217afb3ba74d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:13.535637 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.534505    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjcbb\" (UniqueName: \"kubernetes.io/projected/b0086234-e43e-40f0-be10-217afb3ba74d-kube-api-access-wjcbb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m\" (UID: \"b0086234-e43e-40f0-be10-217afb3ba74d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:13.535637 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.535292    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m\" (UID: \"b0086234-e43e-40f0-be10-217afb3ba74d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:13.535637 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.535606    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m\" (UID: \"b0086234-e43e-40f0-be10-217afb3ba74d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:13.599357 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.599289    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjcbb\" (UniqueName: \"kubernetes.io/projected/b0086234-e43e-40f0-be10-217afb3ba74d-kube-api-access-wjcbb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m\" (UID: \"b0086234-e43e-40f0-be10-217afb3ba74d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:13.622799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.622516    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:13.725613 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.725580    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:13.806139 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.806072    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"]
Nov 02 15:00:13.925779 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:13.925697    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"]
Nov 02 15:00:13.929362 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:13.929332    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b8c1b2_8f94_40f0_a758_f1027a797b69.slice/crio-5ed81ee7bdfe531a25307167695fe6e625bb60d743152cf1897879e84ac25ff6 WatchSource:0}: Error finding container 5ed81ee7bdfe531a25307167695fe6e625bb60d743152cf1897879e84ac25ff6: Status 404 returned error can't find the container with id 5ed81ee7bdfe531a25307167695fe6e625bb60d743152cf1897879e84ac25ff6
Nov 02 15:00:14.031615 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.031530    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"]
Nov 02 15:00:14.034392 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:14.034363    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1624891_a22f_4ed4_8590_f44983e1fbbb.slice/crio-8989bad80181b6d9f83f89a957876f7652f3737baaad473976c6900013a9e5c0 WatchSource:0}: Error finding container 8989bad80181b6d9f83f89a957876f7652f3737baaad473976c6900013a9e5c0: Status 404 returned error can't find the container with id 8989bad80181b6d9f83f89a957876f7652f3737baaad473976c6900013a9e5c0
Nov 02 15:00:14.114782 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.114743    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"]
Nov 02 15:00:14.118180 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:14.118133    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0086234_e43e_40f0_be10_217afb3ba74d.slice/crio-bf5b087f6d0ea004e3458b2c91325398965d67c27a3d545c098d227a63864c0c WatchSource:0}: Error finding container bf5b087f6d0ea004e3458b2c91325398965d67c27a3d545c098d227a63864c0c: Status 404 returned error can't find the container with id bf5b087f6d0ea004e3458b2c91325398965d67c27a3d545c098d227a63864c0c
Nov 02 15:00:14.278568 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:00:14.278535    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:00:14.364852 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.364813    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"]
Nov 02 15:00:14.370727 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.370696    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:14.410457 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.410371    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m" event={"ID":"b0086234-e43e-40f0-be10-217afb3ba74d","Type":"ContainerStarted","Data":"bf5b087f6d0ea004e3458b2c91325398965d67c27a3d545c098d227a63864c0c"}
Nov 02 15:00:14.412136 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.412104    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2" event={"ID":"c0b8c1b2-8f94-40f0-a758-f1027a797b69","Type":"ContainerStarted","Data":"5ed81ee7bdfe531a25307167695fe6e625bb60d743152cf1897879e84ac25ff6"}
Nov 02 15:00:14.413681 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.413619    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr" event={"ID":"e1624891-a22f-4ed4-8590-f44983e1fbbb","Type":"ContainerStarted","Data":"8989bad80181b6d9f83f89a957876f7652f3737baaad473976c6900013a9e5c0"}
Nov 02 15:00:14.415035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.415010    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22" event={"ID":"1c5c0540-2328-49c6-b198-57d221d88c0a","Type":"ContainerStarted","Data":"00702aeed707606f538a542d290a36b4e4327683ac5ce668ae3538d6619eab27"}
Nov 02 15:00:14.429782 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.429719    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"]
Nov 02 15:00:14.440189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.440157    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml\" (UID: \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:14.440389 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.440334    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml\" (UID: \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:14.440389 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.440380    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgtnx\" (UniqueName: \"kubernetes.io/projected/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-kube-api-access-xgtnx\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml\" (UID: \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:14.542380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.541146    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml\" (UID: \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:14.542380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.541288    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml\" (UID: \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:14.542380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.541344    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgtnx\" (UniqueName: \"kubernetes.io/projected/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-kube-api-access-xgtnx\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml\" (UID: \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:14.542380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.542136    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml\" (UID: \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:14.543974 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.543360    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml\" (UID: \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:14.606878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.606841    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgtnx\" (UniqueName: \"kubernetes.io/projected/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-kube-api-access-xgtnx\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml\" (UID: \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:14.691487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:14.691450    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:15.035542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.035470    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"]
Nov 02 15:00:15.040063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.040041    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:15.087438 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.087405    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"]
Nov 02 15:00:15.104408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.104378    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"]
Nov 02 15:00:15.115630 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:15.115582    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1cb9fa4_4a58_459d_9bfc_09d650cf3657.slice/crio-ed8852016977000aa62d9374aa0de6714a7a08110b54285847f96a21e3e69b33 WatchSource:0}: Error finding container ed8852016977000aa62d9374aa0de6714a7a08110b54285847f96a21e3e69b33: Status 404 returned error can't find the container with id ed8852016977000aa62d9374aa0de6714a7a08110b54285847f96a21e3e69b33
Nov 02 15:00:15.147253 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.147015    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4\" (UID: \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:15.147253 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.147079    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mp29\" (UniqueName: \"kubernetes.io/projected/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-kube-api-access-6mp29\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4\" (UID: \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:15.147253 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.147105    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4\" (UID: \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:15.247745 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.247707    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4\" (UID: \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:15.247939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.247765    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mp29\" (UniqueName: \"kubernetes.io/projected/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-kube-api-access-6mp29\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4\" (UID: \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:15.247939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.247784    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4\" (UID: \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:15.248184 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.248164    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4\" (UID: \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:15.248414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.248397    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4\" (UID: \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:15.322642 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.322426    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mp29\" (UniqueName: \"kubernetes.io/projected/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-kube-api-access-6mp29\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4\" (UID: \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:15.363144 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.362875    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:15.430339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.426148    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml" event={"ID":"e1cb9fa4-4a58-459d-9bfc-09d650cf3657","Type":"ContainerStarted","Data":"ed8852016977000aa62d9374aa0de6714a7a08110b54285847f96a21e3e69b33"}
Nov 02 15:00:15.789353 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.789291    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"]
Nov 02 15:00:15.803031 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:15.795765    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf1bed8_3a81_47af_ad2a_d401fc22d0dc.slice/crio-a7fd535325f3278570a65ea7cf38886cd7d4a11f89597a44a94cc423e0590a1a WatchSource:0}: Error finding container a7fd535325f3278570a65ea7cf38886cd7d4a11f89597a44a94cc423e0590a1a: Status 404 returned error can't find the container with id a7fd535325f3278570a65ea7cf38886cd7d4a11f89597a44a94cc423e0590a1a
Nov 02 15:00:15.822991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.821332    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"]
Nov 02 15:00:15.827145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.826624    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.852619 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.852588    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.852783 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.852626    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcd5628-e804-4d09-aa88-9af4a730ef2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.852783 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.852680    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.852783 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.852709    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.852783 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.852754    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.853017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.852798    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.853017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.852822    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.853017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.852843    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c48g6\" (UniqueName: \"kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-kube-api-access-c48g6\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.853017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.852906    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.879488 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.879436    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6x9j8\""
Nov 02 15:00:15.880059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.880030    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 15:00:15.903142 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.903113    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"]
Nov 02 15:00:15.955772 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.955735    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.956126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.955881    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.956126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.955924    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcd5628-e804-4d09-aa88-9af4a730ef2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.956126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.956069    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.956126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.956110    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.956395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.956193    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.956395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.956252    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.956395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.956361    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.956674 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.956398    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c48g6\" (UniqueName: \"kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-kube-api-access-c48g6\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.960165 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.960045    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.963525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.961167    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcd5628-e804-4d09-aa88-9af4a730ef2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.963525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.962423    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.963525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.962837    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.963525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.963290    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.969449 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.969383    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:15.975497 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:15.973353    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:16.025321 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.025273    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c48g6\" (UniqueName: \"kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-kube-api-access-c48g6\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:16.030081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.030052    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwd8j\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:16.147139 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.147054    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"]
Nov 02 15:00:16.149412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.149143    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:16.152457 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.151209    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.214704 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.214675    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-2kx5s\""
Nov 02 15:00:16.233928 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.233845    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"]
Nov 02 15:00:16.262627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.262053    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.262627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.262101    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da03cab-4e6d-43fd-838c-332b9b6beb29-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.262627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.262159    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.262627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.262192    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da03cab-4e6d-43fd-838c-332b9b6beb29-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.262627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.262246    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.262627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.262280    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stkpp\" (UniqueName: \"kubernetes.io/projected/3da03cab-4e6d-43fd-838c-332b9b6beb29-kube-api-access-stkpp\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.262627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.262354    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.262627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.262380    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.262627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.262419    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3da03cab-4e6d-43fd-838c-332b9b6beb29-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.363914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.363873    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3da03cab-4e6d-43fd-838c-332b9b6beb29-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.364110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.363945    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.364110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.363978    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da03cab-4e6d-43fd-838c-332b9b6beb29-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.364110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.364031    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.364110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.364061    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da03cab-4e6d-43fd-838c-332b9b6beb29-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.364723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.364112    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.364723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.364143    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stkpp\" (UniqueName: \"kubernetes.io/projected/3da03cab-4e6d-43fd-838c-332b9b6beb29-kube-api-access-stkpp\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.364723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.364206    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.364723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.364232    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.365046 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.365000    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.365609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.365584    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.366232 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.366209    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3da03cab-4e6d-43fd-838c-332b9b6beb29-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.366505 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.366484    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.367264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.367213    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.370485 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.370443    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3da03cab-4e6d-43fd-838c-332b9b6beb29-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.400251 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.400179    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3da03cab-4e6d-43fd-838c-332b9b6beb29-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.436095 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.436054    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3da03cab-4e6d-43fd-838c-332b9b6beb29-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.438714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.438628    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stkpp\" (UniqueName: \"kubernetes.io/projected/3da03cab-4e6d-43fd-838c-332b9b6beb29-kube-api-access-stkpp\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p\" (UID: \"3da03cab-4e6d-43fd-838c-332b9b6beb29\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.446768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.446687    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4" event={"ID":"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc","Type":"ContainerStarted","Data":"a7fd535325f3278570a65ea7cf38886cd7d4a11f89597a44a94cc423e0590a1a"}
Nov 02 15:00:16.470892 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.470610    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:16.570055 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.570025    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"]
Nov 02 15:00:16.580636 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:16.580582    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcd5628_e804_4d09_aa88_9af4a730ef2c.slice/crio-0091a7ebb0b327094e15d5f65cbbbf8e52397142957ebdf7f0d0fb70059fea96 WatchSource:0}: Error finding container 0091a7ebb0b327094e15d5f65cbbbf8e52397142957ebdf7f0d0fb70059fea96: Status 404 returned error can't find the container with id 0091a7ebb0b327094e15d5f65cbbbf8e52397142957ebdf7f0d0fb70059fea96
Nov 02 15:00:16.893177 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:16.892898    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"]
Nov 02 15:00:17.468904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:17.468803    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" event={"ID":"adcd5628-e804-4d09-aa88-9af4a730ef2c","Type":"ContainerStarted","Data":"0091a7ebb0b327094e15d5f65cbbbf8e52397142957ebdf7f0d0fb70059fea96"}
Nov 02 15:00:17.472098 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:17.472035    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p" event={"ID":"3da03cab-4e6d-43fd-838c-332b9b6beb29","Type":"ContainerStarted","Data":"a8b6b55154c36518614a6950f5ab1b38092e94a54205c18e3aba9ca4a4d78a0b"}
Nov 02 15:00:18.487170 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:18.487096    4891 generic.go:356] "Generic (PLEG): container finished" podID="1f934255-4e4b-4d05-91a7-15ce08f174e4" containerID="c77afa715344e47f7dbbc1e9932029f1ab732f3271baf27a98348587fc9d0e25" exitCode=0
Nov 02 15:00:18.487724 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:18.487204    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f" event={"ID":"1f934255-4e4b-4d05-91a7-15ce08f174e4","Type":"ContainerDied","Data":"c77afa715344e47f7dbbc1e9932029f1ab732f3271baf27a98348587fc9d0e25"}
Nov 02 15:00:20.511232 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:20.511149    4891 generic.go:356] "Generic (PLEG): container finished" podID="1f934255-4e4b-4d05-91a7-15ce08f174e4" containerID="10d47feaa84eb16fb0d2efa905b502a6c13334a45a7eed8b11d8a12fa320d6df" exitCode=0
Nov 02 15:00:20.511724 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:20.511276    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f" event={"ID":"1f934255-4e4b-4d05-91a7-15ce08f174e4","Type":"ContainerDied","Data":"10d47feaa84eb16fb0d2efa905b502a6c13334a45a7eed8b11d8a12fa320d6df"}
Nov 02 15:00:20.524108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:20.522599    4891 generic.go:356] "Generic (PLEG): container finished" podID="6655aa2e-f88a-45f2-a5d7-927a9802ab4c" containerID="abae85ff8905b5cda046825a2d9050d9ef0d5b999968b235e5506cf7532b6fe4" exitCode=0
Nov 02 15:00:20.526723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:20.526688    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2" event={"ID":"6655aa2e-f88a-45f2-a5d7-927a9802ab4c","Type":"ContainerDied","Data":"abae85ff8905b5cda046825a2d9050d9ef0d5b999968b235e5506cf7532b6fe4"}
Nov 02 15:00:25.590882 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:25.590847    4891 generic.go:356] "Generic (PLEG): container finished" podID="1c5c0540-2328-49c6-b198-57d221d88c0a" containerID="411656daa4337a0dd7adeac86cf92387a89c9065e33b5db78aba120538ea27c9" exitCode=0
Nov 02 15:00:25.591303 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:25.590926    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22" event={"ID":"1c5c0540-2328-49c6-b198-57d221d88c0a","Type":"ContainerDied","Data":"411656daa4337a0dd7adeac86cf92387a89c9065e33b5db78aba120538ea27c9"}
Nov 02 15:00:26.603822 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:26.603780    4891 generic.go:356] "Generic (PLEG): container finished" podID="b0086234-e43e-40f0-be10-217afb3ba74d" containerID="5580e06382d0efb79eef684497197ba1a9b2a4ec234d1252e68b7126bae38b14" exitCode=0
Nov 02 15:00:26.604477 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:26.603871    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m" event={"ID":"b0086234-e43e-40f0-be10-217afb3ba74d","Type":"ContainerDied","Data":"5580e06382d0efb79eef684497197ba1a9b2a4ec234d1252e68b7126bae38b14"}
Nov 02 15:00:26.609194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:26.609137    4891 generic.go:356] "Generic (PLEG): container finished" podID="c0b8c1b2-8f94-40f0-a758-f1027a797b69" containerID="f1ae665d81a5c2300618a354dbe6dc4791f45599e4573f7acc09ed2856708fdd" exitCode=0
Nov 02 15:00:26.609341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:26.609239    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2" event={"ID":"c0b8c1b2-8f94-40f0-a758-f1027a797b69","Type":"ContainerDied","Data":"f1ae665d81a5c2300618a354dbe6dc4791f45599e4573f7acc09ed2856708fdd"}
Nov 02 15:00:26.615873 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:26.615208    4891 generic.go:356] "Generic (PLEG): container finished" podID="e1624891-a22f-4ed4-8590-f44983e1fbbb" containerID="4235d688aa959806746e3c5abe83e3b762404da37097fd992712d75e158bec64" exitCode=0
Nov 02 15:00:26.615873 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:26.615296    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr" event={"ID":"e1624891-a22f-4ed4-8590-f44983e1fbbb","Type":"ContainerDied","Data":"4235d688aa959806746e3c5abe83e3b762404da37097fd992712d75e158bec64"}
Nov 02 15:00:27.048717 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.048673    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:00:27.048832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.048770    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:00:27.048832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.048814    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:00:27.270622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.269252    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:00:27.270622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.269376    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:00:27.270622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.269433    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:00:27.622073 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.622042    4891 generic.go:356] "Generic (PLEG): container finished" podID="f3b70660-5510-4dd2-9e89-83cc2193fc93" containerID="19939356bdcdda93f707de07b67a623a89ef78b62e6c62164d4763cb55389539" exitCode=0
Nov 02 15:00:27.622455 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.622163    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x" event={"ID":"f3b70660-5510-4dd2-9e89-83cc2193fc93","Type":"ContainerDied","Data":"19939356bdcdda93f707de07b67a623a89ef78b62e6c62164d4763cb55389539"}
Nov 02 15:00:27.624451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.624428    4891 generic.go:356] "Generic (PLEG): container finished" podID="e1cb9fa4-4a58-459d-9bfc-09d650cf3657" containerID="5d320366fc80b417138bb419f6fa7c2bda01325927ddccd1d0505856b1e82af8" exitCode=0
Nov 02 15:00:27.624724 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.624648    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml" event={"ID":"e1cb9fa4-4a58-459d-9bfc-09d650cf3657","Type":"ContainerDied","Data":"5d320366fc80b417138bb419f6fa7c2bda01325927ddccd1d0505856b1e82af8"}
Nov 02 15:00:27.627119 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.627081    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p" event={"ID":"3da03cab-4e6d-43fd-838c-332b9b6beb29","Type":"ContainerStarted","Data":"8eb68a64c77b6afe8c597f1a95e738aa2147cf4e999113283eba82e8417b9618"}
Nov 02 15:00:27.628787 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.628764    4891 generic.go:356] "Generic (PLEG): container finished" podID="1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" containerID="d6e0c72ae85f2ed8deeb44474e81b7f4ca8a40607a26c7bb77d806540faa64a3" exitCode=0
Nov 02 15:00:27.628878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.628856    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4" event={"ID":"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc","Type":"ContainerDied","Data":"d6e0c72ae85f2ed8deeb44474e81b7f4ca8a40607a26c7bb77d806540faa64a3"}
Nov 02 15:00:27.630933 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.630877    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" event={"ID":"adcd5628-e804-4d09-aa88-9af4a730ef2c","Type":"ContainerStarted","Data":"540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de"}
Nov 02 15:00:27.868183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:27.868131    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" podStartSLOduration=2.184518909 podStartE2EDuration="12.868115238s" totalImagesPullingTime="10.683596329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:27.866881277 +0000 UTC m=+235.459498077" watchObservedRunningTime="2026-11-02 15:00:27.868115238 +0000 UTC m=+235.460732037"
Nov 02 15:00:28.150143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.150120    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:28.155092 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.155060    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:28.265192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.265132    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p" podStartSLOduration=2.11675893 podStartE2EDuration="12.26511257s" totalImagesPullingTime="10.14835364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:28.261624621 +0000 UTC m=+235.854241420" watchObservedRunningTime="2026-11-02 15:00:28.26511257 +0000 UTC m=+235.857729370"
Nov 02 15:00:28.471596 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.471571    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:28.480514 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.477023    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:28.638733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.638646    4891 generic.go:356] "Generic (PLEG): container finished" podID="6655aa2e-f88a-45f2-a5d7-927a9802ab4c" containerID="a961ac521a2b94b1fdd49ffd6c680163af56e8fac766be9c9e58a9e0a927d8c1" exitCode=0
Nov 02 15:00:28.639146 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.638779    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2" event={"ID":"6655aa2e-f88a-45f2-a5d7-927a9802ab4c","Type":"ContainerDied","Data":"a961ac521a2b94b1fdd49ffd6c680163af56e8fac766be9c9e58a9e0a927d8c1"}
Nov 02 15:00:28.642080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.642046    4891 generic.go:356] "Generic (PLEG): container finished" podID="e1cb9fa4-4a58-459d-9bfc-09d650cf3657" containerID="5a169421d88898344a894d95c814a9bfc9eeb1db073bb9a14bb996dc121963ce" exitCode=0
Nov 02 15:00:28.642199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.642180    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml" event={"ID":"e1cb9fa4-4a58-459d-9bfc-09d650cf3657","Type":"ContainerDied","Data":"5a169421d88898344a894d95c814a9bfc9eeb1db073bb9a14bb996dc121963ce"}
Nov 02 15:00:28.646723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.646698    4891 generic.go:356] "Generic (PLEG): container finished" podID="b0086234-e43e-40f0-be10-217afb3ba74d" containerID="68b1b28fde4ee08857fae01a137ae7e7042b99f3bc28a7f77a1c31332191ac20" exitCode=0
Nov 02 15:00:28.646834 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.646770    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m" event={"ID":"b0086234-e43e-40f0-be10-217afb3ba74d","Type":"ContainerDied","Data":"68b1b28fde4ee08857fae01a137ae7e7042b99f3bc28a7f77a1c31332191ac20"}
Nov 02 15:00:28.649178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.649044    4891 generic.go:356] "Generic (PLEG): container finished" podID="c0b8c1b2-8f94-40f0-a758-f1027a797b69" containerID="75906c9f91c49ec723ec95ec082f2bbd3f62399f8c9d6bda3a5e6d4993d79f74" exitCode=0
Nov 02 15:00:28.649178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.649131    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2" event={"ID":"c0b8c1b2-8f94-40f0-a758-f1027a797b69","Type":"ContainerDied","Data":"75906c9f91c49ec723ec95ec082f2bbd3f62399f8c9d6bda3a5e6d4993d79f74"}
Nov 02 15:00:28.651722 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.651646    4891 generic.go:356] "Generic (PLEG): container finished" podID="1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" containerID="8667f9761e2d147663ad71df85d102cc5997bafc1cb16930cddfb97967afbd77" exitCode=0
Nov 02 15:00:28.651722 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.651667    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4" event={"ID":"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc","Type":"ContainerDied","Data":"8667f9761e2d147663ad71df85d102cc5997bafc1cb16930cddfb97967afbd77"}
Nov 02 15:00:28.652449 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.652355    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:28.652449 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.652393    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:28.653298 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.653277    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:00:28.653608 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:28.653592    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p"
Nov 02 15:00:29.662837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:29.662802    4891 generic.go:356] "Generic (PLEG): container finished" podID="e1624891-a22f-4ed4-8590-f44983e1fbbb" containerID="58a0c535ad67b6d73269334b5b00e33e0e6d9621a7767516fe7223ddf4d4d807" exitCode=0
Nov 02 15:00:29.663331 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:29.662934    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr" event={"ID":"e1624891-a22f-4ed4-8590-f44983e1fbbb","Type":"ContainerDied","Data":"58a0c535ad67b6d73269334b5b00e33e0e6d9621a7767516fe7223ddf4d4d807"}
Nov 02 15:00:29.670530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:29.668743    4891 generic.go:356] "Generic (PLEG): container finished" podID="1c5c0540-2328-49c6-b198-57d221d88c0a" containerID="0391f8747ac98ea7e9018edc5de83525b2507fa7bf15cc6e7a8e5900c4807bc1" exitCode=0
Nov 02 15:00:29.670530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:29.669967    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22" event={"ID":"1c5c0540-2328-49c6-b198-57d221d88c0a","Type":"ContainerDied","Data":"0391f8747ac98ea7e9018edc5de83525b2507fa7bf15cc6e7a8e5900c4807bc1"}
Nov 02 15:00:35.736279 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:35.736205    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x" event={"ID":"f3b70660-5510-4dd2-9e89-83cc2193fc93","Type":"ContainerStarted","Data":"8f40baf3ff5eccd6f9925cc4bc2478c2b40310b243311ed71133b296193a794c"}
Nov 02 15:00:36.754841 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:36.753813    4891 generic.go:356] "Generic (PLEG): container finished" podID="f3b70660-5510-4dd2-9e89-83cc2193fc93" containerID="8f40baf3ff5eccd6f9925cc4bc2478c2b40310b243311ed71133b296193a794c" exitCode=0
Nov 02 15:00:36.754841 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:36.753899    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x" event={"ID":"f3b70660-5510-4dd2-9e89-83cc2193fc93","Type":"ContainerDied","Data":"8f40baf3ff5eccd6f9925cc4bc2478c2b40310b243311ed71133b296193a794c"}
Nov 02 15:00:39.788576 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:39.788538    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml" event={"ID":"e1cb9fa4-4a58-459d-9bfc-09d650cf3657","Type":"ContainerStarted","Data":"70bc6d5a94ade0383477bf04cd31b5b35e596a8f6438c2cb3b1071b5fd95fab5"}
Nov 02 15:00:39.792175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:39.792146    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m" event={"ID":"b0086234-e43e-40f0-be10-217afb3ba74d","Type":"ContainerStarted","Data":"76dcb4f728ddf66b2e17c06474f99df66d08f7da01ec7df068ae849bd721f619"}
Nov 02 15:00:39.795941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:39.795920    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2" event={"ID":"c0b8c1b2-8f94-40f0-a758-f1027a797b69","Type":"ContainerStarted","Data":"9ffd96d53d827d83f9c0273932dc8cf6aa48b8effff27a32d04836027e00e218"}
Nov 02 15:00:39.800359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:39.800335    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4" event={"ID":"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc","Type":"ContainerStarted","Data":"6364db326ee74c6bd8aaf1d997b2bebe50a9ab1b3357340f972b343b8f73286c"}
Nov 02 15:00:39.805251 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:39.805224    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f" event={"ID":"1f934255-4e4b-4d05-91a7-15ce08f174e4","Type":"ContainerStarted","Data":"4be60b90816c6fad1385d7096c0f61df6769e0e2ab703c95d5ade0ab520018b1"}
Nov 02 15:00:39.810088 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:39.810063    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2" event={"ID":"6655aa2e-f88a-45f2-a5d7-927a9802ab4c","Type":"ContainerStarted","Data":"b40f593cea2845a105a305037d04d7af898ce4abf6c9e67bd41f77d03aeaeda9"}
Nov 02 15:00:39.915023 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:39.914967    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml" podStartSLOduration=2.9094018249999998 podStartE2EDuration="25.914953041s" totalImagesPullingTime="23.005551216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:39.913971985 +0000 UTC m=+247.506588795" watchObservedRunningTime="2026-11-02 15:00:39.914953041 +0000 UTC m=+247.507569923"
Nov 02 15:00:40.041982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.040713    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2" podStartSLOduration=2.646130365 podStartE2EDuration="27.040694461s" totalImagesPullingTime="24.394564096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:40.039158269 +0000 UTC m=+247.631775068" watchObservedRunningTime="2026-11-02 15:00:40.040694461 +0000 UTC m=+247.633311262"
Nov 02 15:00:40.160061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.160002    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m" podStartSLOduration=3.084840364 podStartE2EDuration="27.159981649s" totalImagesPullingTime="24.075141285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:40.157639693 +0000 UTC m=+247.750256494" watchObservedRunningTime="2026-11-02 15:00:40.159981649 +0000 UTC m=+247.752598451"
Nov 02 15:00:40.399423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.397425    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2" podStartSLOduration=4.383211469 podStartE2EDuration="33.397405385s" totalImagesPullingTime="29.014193916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:40.396786932 +0000 UTC m=+247.989403735" watchObservedRunningTime="2026-11-02 15:00:40.397405385 +0000 UTC m=+247.990022185"
Nov 02 15:00:40.516814 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.516758    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4" podStartSLOduration=3.922708012 podStartE2EDuration="26.516743723s" totalImagesPullingTime="22.594035711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:40.515741989 +0000 UTC m=+248.108358803" watchObservedRunningTime="2026-11-02 15:00:40.516743723 +0000 UTC m=+248.109360522"
Nov 02 15:00:40.818734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.818698    4891 generic.go:356] "Generic (PLEG): container finished" podID="6655aa2e-f88a-45f2-a5d7-927a9802ab4c" containerID="b40f593cea2845a105a305037d04d7af898ce4abf6c9e67bd41f77d03aeaeda9" exitCode=0
Nov 02 15:00:40.819236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.818791    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2" event={"ID":"6655aa2e-f88a-45f2-a5d7-927a9802ab4c","Type":"ContainerDied","Data":"b40f593cea2845a105a305037d04d7af898ce4abf6c9e67bd41f77d03aeaeda9"}
Nov 02 15:00:40.820790 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.820772    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:00:40.821484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.821459    4891 generic.go:356] "Generic (PLEG): container finished" podID="e1cb9fa4-4a58-459d-9bfc-09d650cf3657" containerID="70bc6d5a94ade0383477bf04cd31b5b35e596a8f6438c2cb3b1071b5fd95fab5" exitCode=1
Nov 02 15:00:40.821595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.821534    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml" event={"ID":"e1cb9fa4-4a58-459d-9bfc-09d650cf3657","Type":"ContainerDied","Data":"70bc6d5a94ade0383477bf04cd31b5b35e596a8f6438c2cb3b1071b5fd95fab5"}
Nov 02 15:00:40.823891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.823869    4891 generic.go:356] "Generic (PLEG): container finished" podID="b0086234-e43e-40f0-be10-217afb3ba74d" containerID="76dcb4f728ddf66b2e17c06474f99df66d08f7da01ec7df068ae849bd721f619" exitCode=0
Nov 02 15:00:40.824011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.823937    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m" event={"ID":"b0086234-e43e-40f0-be10-217afb3ba74d","Type":"ContainerDied","Data":"76dcb4f728ddf66b2e17c06474f99df66d08f7da01ec7df068ae849bd721f619"}
Nov 02 15:00:40.826256 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.826232    4891 generic.go:356] "Generic (PLEG): container finished" podID="c0b8c1b2-8f94-40f0-a758-f1027a797b69" containerID="9ffd96d53d827d83f9c0273932dc8cf6aa48b8effff27a32d04836027e00e218" exitCode=0
Nov 02 15:00:40.826382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.826332    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2" event={"ID":"c0b8c1b2-8f94-40f0-a758-f1027a797b69","Type":"ContainerDied","Data":"9ffd96d53d827d83f9c0273932dc8cf6aa48b8effff27a32d04836027e00e218"}
Nov 02 15:00:40.828515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.828494    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr" event={"ID":"e1624891-a22f-4ed4-8590-f44983e1fbbb","Type":"ContainerStarted","Data":"48f452586d65ff5e69038b8ba8b26f104594a955cfb5bddcc3e294d85e6065ea"}
Nov 02 15:00:40.830243 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.830225    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:00:40.831044 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.830917    4891 generic.go:356] "Generic (PLEG): container finished" podID="1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" containerID="6364db326ee74c6bd8aaf1d997b2bebe50a9ab1b3357340f972b343b8f73286c" exitCode=1
Nov 02 15:00:40.831044 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.830998    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4" event={"ID":"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc","Type":"ContainerDied","Data":"6364db326ee74c6bd8aaf1d997b2bebe50a9ab1b3357340f972b343b8f73286c"}
Nov 02 15:00:40.833347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.833298    4891 generic.go:356] "Generic (PLEG): container finished" podID="1f934255-4e4b-4d05-91a7-15ce08f174e4" containerID="4be60b90816c6fad1385d7096c0f61df6769e0e2ab703c95d5ade0ab520018b1" exitCode=0
Nov 02 15:00:40.833436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.833345    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f" event={"ID":"1f934255-4e4b-4d05-91a7-15ce08f174e4","Type":"ContainerDied","Data":"4be60b90816c6fad1385d7096c0f61df6769e0e2ab703c95d5ade0ab520018b1"}
Nov 02 15:00:40.835810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.835785    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22" event={"ID":"1c5c0540-2328-49c6-b198-57d221d88c0a","Type":"ContainerStarted","Data":"b9f466e4f816949e039c5e906bd911055b758ddd07af7ad2670f9bce8004c5d9"}
Nov 02 15:00:40.963665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:40.963643    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:41.061996 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.061943    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22" podStartSLOduration=2.598306787 podStartE2EDuration="28.061927587s" totalImagesPullingTime="25.4636208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:41.059449441 +0000 UTC m=+248.652066242" watchObservedRunningTime="2026-11-02 15:00:41.061927587 +0000 UTC m=+248.654544383"
Nov 02 15:00:41.119714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.119677    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-util\" (UniqueName: \"kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-util\") pod \"1f934255-4e4b-4d05-91a7-15ce08f174e4\" (UID: \"1f934255-4e4b-4d05-91a7-15ce08f174e4\") "
Nov 02 15:00:41.119894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.119769    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f934255-4e4b-4d05-91a7-15ce08f174e4-kube-api-access-v8ht8\" (UniqueName: \"kubernetes.io/projected/1f934255-4e4b-4d05-91a7-15ce08f174e4-kube-api-access-v8ht8\") pod \"1f934255-4e4b-4d05-91a7-15ce08f174e4\" (UID: \"1f934255-4e4b-4d05-91a7-15ce08f174e4\") "
Nov 02 15:00:41.119894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.119852    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-bundle\") pod \"1f934255-4e4b-4d05-91a7-15ce08f174e4\" (UID: \"1f934255-4e4b-4d05-91a7-15ce08f174e4\") "
Nov 02 15:00:41.120650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.120612    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-bundle" pod "1f934255-4e4b-4d05-91a7-15ce08f174e4" (UID: "1f934255-4e4b-4d05-91a7-15ce08f174e4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:41.122557 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.122525    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f934255-4e4b-4d05-91a7-15ce08f174e4-kube-api-access-v8ht8" pod "1f934255-4e4b-4d05-91a7-15ce08f174e4" (UID: "1f934255-4e4b-4d05-91a7-15ce08f174e4"). InnerVolumeSpecName "kube-api-access-v8ht8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:41.126694 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.126666    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-util" pod "1f934255-4e4b-4d05-91a7-15ce08f174e4" (UID: "1f934255-4e4b-4d05-91a7-15ce08f174e4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:41.220826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.220790    4891 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:41.220826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.220827    4891 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f934255-4e4b-4d05-91a7-15ce08f174e4-util\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:41.221014 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.220842    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ht8\" (UniqueName: \"kubernetes.io/projected/1f934255-4e4b-4d05-91a7-15ce08f174e4-kube-api-access-v8ht8\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:41.297875 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.297812    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr" podStartSLOduration=3.76727963 podStartE2EDuration="28.2977953s" totalImagesPullingTime="24.53051567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:41.296982556 +0000 UTC m=+248.889599355" watchObservedRunningTime="2026-11-02 15:00:41.2977953 +0000 UTC m=+248.890412102"
Nov 02 15:00:41.845586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.845546    4891 generic.go:356] "Generic (PLEG): container finished" podID="e1624891-a22f-4ed4-8590-f44983e1fbbb" containerID="48f452586d65ff5e69038b8ba8b26f104594a955cfb5bddcc3e294d85e6065ea" exitCode=0
Nov 02 15:00:41.846090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.845634    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr" event={"ID":"e1624891-a22f-4ed4-8590-f44983e1fbbb","Type":"ContainerDied","Data":"48f452586d65ff5e69038b8ba8b26f104594a955cfb5bddcc3e294d85e6065ea"}
Nov 02 15:00:41.847820 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.847798    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f" event={"ID":"1f934255-4e4b-4d05-91a7-15ce08f174e4","Type":"ContainerDied","Data":"b811ea97262b73966d726cb893a09382c806eadc2b98528190f1dea0555b6b6c"}
Nov 02 15:00:41.847945 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.847838    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b811ea97262b73966d726cb893a09382c806eadc2b98528190f1dea0555b6b6c"
Nov 02 15:00:41.847945 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.847807    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xw52f"
Nov 02 15:00:41.850237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.850212    4891 generic.go:356] "Generic (PLEG): container finished" podID="1c5c0540-2328-49c6-b198-57d221d88c0a" containerID="b9f466e4f816949e039c5e906bd911055b758ddd07af7ad2670f9bce8004c5d9" exitCode=0
Nov 02 15:00:41.850372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:41.850301    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22" event={"ID":"1c5c0540-2328-49c6-b198-57d221d88c0a","Type":"ContainerDied","Data":"b9f466e4f816949e039c5e906bd911055b758ddd07af7ad2670f9bce8004c5d9"}
Nov 02 15:00:42.023055 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.023032    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:00:42.025934 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.025908    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:42.131432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.130388    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-kube-api-access-6mp29\" (UniqueName: \"kubernetes.io/projected/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-kube-api-access-6mp29\") pod \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\" (UID: \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\") "
Nov 02 15:00:42.131432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.130543    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-bundle\") pod \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\" (UID: \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\") "
Nov 02 15:00:42.131432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.130624    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-util\" (UniqueName: \"kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-util\") pod \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\" (UID: \"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc\") "
Nov 02 15:00:42.132040 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.131994    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-bundle" pod "1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" (UID: "1cf1bed8-3a81-47af-ad2a-d401fc22d0dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:42.133423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.133396    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-kube-api-access-6mp29" pod "1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" (UID: "1cf1bed8-3a81-47af-ad2a-d401fc22d0dc"). InnerVolumeSpecName "kube-api-access-6mp29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:42.137057 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.136941    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-util" pod "1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" (UID: "1cf1bed8-3a81-47af-ad2a-d401fc22d0dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:42.162262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.162234    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:42.164505 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.164476    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:00:42.166213 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.166189    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:42.169117 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.169093    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:42.173485 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.173465    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:42.231460 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.231427    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-util\" (UniqueName: \"kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-util\") pod \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\" (UID: \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\") "
Nov 02 15:00:42.231641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.231474    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-bundle\") pod \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\" (UID: \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\") "
Nov 02 15:00:42.231641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.231522    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-kube-api-access-xgtnx\" (UniqueName: \"kubernetes.io/projected/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-kube-api-access-xgtnx\") pod \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\" (UID: \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\") "
Nov 02 15:00:42.231641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.231556    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-kube-api-access-7tdkg\" (UniqueName: \"kubernetes.io/projected/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-kube-api-access-7tdkg\") pod \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\" (UID: \"6655aa2e-f88a-45f2-a5d7-927a9802ab4c\") "
Nov 02 15:00:42.231641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.231627    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-util\" (UniqueName: \"kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-util\") pod \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\" (UID: \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\") "
Nov 02 15:00:42.231863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.231706    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-bundle\") pod \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\" (UID: \"e1cb9fa4-4a58-459d-9bfc-09d650cf3657\") "
Nov 02 15:00:42.231949 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.231932    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mp29\" (UniqueName: \"kubernetes.io/projected/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-kube-api-access-6mp29\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.232004 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.231954    4891 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.232004 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.231970    4891 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cf1bed8-3a81-47af-ad2a-d401fc22d0dc-util\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.232387 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.232350    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-bundle" pod "e1cb9fa4-4a58-459d-9bfc-09d650cf3657" (UID: "e1cb9fa4-4a58-459d-9bfc-09d650cf3657"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:42.232680 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.232657    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-bundle" pod "6655aa2e-f88a-45f2-a5d7-927a9802ab4c" (UID: "6655aa2e-f88a-45f2-a5d7-927a9802ab4c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:42.234337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.234289    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-kube-api-access-xgtnx" pod "e1cb9fa4-4a58-459d-9bfc-09d650cf3657" (UID: "e1cb9fa4-4a58-459d-9bfc-09d650cf3657"). InnerVolumeSpecName "kube-api-access-xgtnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:42.234434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.234299    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-kube-api-access-7tdkg" pod "6655aa2e-f88a-45f2-a5d7-927a9802ab4c" (UID: "6655aa2e-f88a-45f2-a5d7-927a9802ab4c"). InnerVolumeSpecName "kube-api-access-7tdkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:42.237850 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.237825    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-util" pod "6655aa2e-f88a-45f2-a5d7-927a9802ab4c" (UID: "6655aa2e-f88a-45f2-a5d7-927a9802ab4c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:42.238518 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.238489    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-util" pod "e1cb9fa4-4a58-459d-9bfc-09d650cf3657" (UID: "e1cb9fa4-4a58-459d-9bfc-09d650cf3657"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:42.332901 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.332870    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-util\" (UniqueName: \"kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-util\") pod \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\" (UID: \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\") "
Nov 02 15:00:42.333073 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.332958    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b8c1b2-8f94-40f0-a758-f1027a797b69-kube-api-access-zd8w7\" (UniqueName: \"kubernetes.io/projected/c0b8c1b2-8f94-40f0-a758-f1027a797b69-kube-api-access-zd8w7\") pod \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\" (UID: \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\") "
Nov 02 15:00:42.333073 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.332985    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-bundle\") pod \"b0086234-e43e-40f0-be10-217afb3ba74d\" (UID: \"b0086234-e43e-40f0-be10-217afb3ba74d\") "
Nov 02 15:00:42.333073 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.333012    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0086234-e43e-40f0-be10-217afb3ba74d-kube-api-access-wjcbb\" (UniqueName: \"kubernetes.io/projected/b0086234-e43e-40f0-be10-217afb3ba74d-kube-api-access-wjcbb\") pod \"b0086234-e43e-40f0-be10-217afb3ba74d\" (UID: \"b0086234-e43e-40f0-be10-217afb3ba74d\") "
Nov 02 15:00:42.333073 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.333036    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-bundle\") pod \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\" (UID: \"c0b8c1b2-8f94-40f0-a758-f1027a797b69\") "
Nov 02 15:00:42.333073 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.333055    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-util\" (UniqueName: \"kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-util\") pod \"b0086234-e43e-40f0-be10-217afb3ba74d\" (UID: \"b0086234-e43e-40f0-be10-217afb3ba74d\") "
Nov 02 15:00:42.333376 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.333240    4891 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-util\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.333376 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.333259    4891 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.333376 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.333273    4891 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-util\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.333376 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.333285    4891 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.333376 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.333299    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgtnx\" (UniqueName: \"kubernetes.io/projected/e1cb9fa4-4a58-459d-9bfc-09d650cf3657-kube-api-access-xgtnx\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.333376 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.333336    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tdkg\" (UniqueName: \"kubernetes.io/projected/6655aa2e-f88a-45f2-a5d7-927a9802ab4c-kube-api-access-7tdkg\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.333769 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.333740    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-bundle" pod "c0b8c1b2-8f94-40f0-a758-f1027a797b69" (UID: "c0b8c1b2-8f94-40f0-a758-f1027a797b69"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:42.335942 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.335909    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b8c1b2-8f94-40f0-a758-f1027a797b69-kube-api-access-zd8w7" pod "c0b8c1b2-8f94-40f0-a758-f1027a797b69" (UID: "c0b8c1b2-8f94-40f0-a758-f1027a797b69"). InnerVolumeSpecName "kube-api-access-zd8w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:42.336073 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.336024    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-bundle" pod "b0086234-e43e-40f0-be10-217afb3ba74d" (UID: "b0086234-e43e-40f0-be10-217afb3ba74d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:42.336148 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.336101    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0086234-e43e-40f0-be10-217afb3ba74d-kube-api-access-wjcbb" pod "b0086234-e43e-40f0-be10-217afb3ba74d" (UID: "b0086234-e43e-40f0-be10-217afb3ba74d"). InnerVolumeSpecName "kube-api-access-wjcbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:42.339047 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.339021    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-util" pod "b0086234-e43e-40f0-be10-217afb3ba74d" (UID: "b0086234-e43e-40f0-be10-217afb3ba74d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:42.340347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.340323    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-util" pod "c0b8c1b2-8f94-40f0-a758-f1027a797b69" (UID: "c0b8c1b2-8f94-40f0-a758-f1027a797b69"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:42.433835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.433746    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd8w7\" (UniqueName: \"kubernetes.io/projected/c0b8c1b2-8f94-40f0-a758-f1027a797b69-kube-api-access-zd8w7\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.433835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.433783    4891 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.433835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.433797    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjcbb\" (UniqueName: \"kubernetes.io/projected/b0086234-e43e-40f0-be10-217afb3ba74d-kube-api-access-wjcbb\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.433835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.433810    4891 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.433835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.433825    4891 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0086234-e43e-40f0-be10-217afb3ba74d-util\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.433835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.433838    4891 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b8c1b2-8f94-40f0-a758-f1027a797b69-util\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:42.856947 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.856922    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2"
Nov 02 15:00:42.857418 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.856921    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpmpg2" event={"ID":"6655aa2e-f88a-45f2-a5d7-927a9802ab4c","Type":"ContainerDied","Data":"e1179b03cbf2261bb34b66c018aedf85fcdb60857b42bf9fb4c3cab3ff826da8"}
Nov 02 15:00:42.857418 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.857048    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1179b03cbf2261bb34b66c018aedf85fcdb60857b42bf9fb4c3cab3ff826da8"
Nov 02 15:00:42.859234 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.859208    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x" event={"ID":"f3b70660-5510-4dd2-9e89-83cc2193fc93","Type":"ContainerStarted","Data":"9a21264cddc807036d2f257c41a7249a38af222418d0096884f5506191b297aa"}
Nov 02 15:00:42.860559 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.860541    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:00:42.861228 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.861207    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml"
Nov 02 15:00:42.861351 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.861213    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml" event={"ID":"e1cb9fa4-4a58-459d-9bfc-09d650cf3657","Type":"ContainerDied","Data":"ed8852016977000aa62d9374aa0de6714a7a08110b54285847f96a21e3e69b33"}
Nov 02 15:00:42.861351 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.861336    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed8852016977000aa62d9374aa0de6714a7a08110b54285847f96a21e3e69b33"
Nov 02 15:00:42.863147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.863129    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m" event={"ID":"b0086234-e43e-40f0-be10-217afb3ba74d","Type":"ContainerDied","Data":"bf5b087f6d0ea004e3458b2c91325398965d67c27a3d545c098d227a63864c0c"}
Nov 02 15:00:42.863247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.863163    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf5b087f6d0ea004e3458b2c91325398965d67c27a3d545c098d227a63864c0c"
Nov 02 15:00:42.863247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.863191    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axch4m"
Nov 02 15:00:42.865153 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.865137    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2"
Nov 02 15:00:42.865252 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.865147    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dggw2" event={"ID":"c0b8c1b2-8f94-40f0-a758-f1027a797b69","Type":"ContainerDied","Data":"5ed81ee7bdfe531a25307167695fe6e625bb60d743152cf1897879e84ac25ff6"}
Nov 02 15:00:42.865252 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.865209    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ed81ee7bdfe531a25307167695fe6e625bb60d743152cf1897879e84ac25ff6"
Nov 02 15:00:42.866500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.866480    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:00:42.867227 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.867207    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4" event={"ID":"1cf1bed8-3a81-47af-ad2a-d401fc22d0dc","Type":"ContainerDied","Data":"a7fd535325f3278570a65ea7cf38886cd7d4a11f89597a44a94cc423e0590a1a"}
Nov 02 15:00:42.867227 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.867225    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4"
Nov 02 15:00:42.867386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.867245    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7fd535325f3278570a65ea7cf38886cd7d4a11f89597a44a94cc423e0590a1a"
Nov 02 15:00:42.979483 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.978469    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x" podStartSLOduration=2.791056204 podStartE2EDuration="32.978451274s" totalImagesPullingTime="30.18739507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:42.978006496 +0000 UTC m=+250.570623296" watchObservedRunningTime="2026-11-02 15:00:42.978451274 +0000 UTC m=+250.571068071"
Nov 02 15:00:42.985576 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:42.985546    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:43.013223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.013195    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:43.138849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.138812    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-util\" (UniqueName: \"kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-util\") pod \"e1624891-a22f-4ed4-8590-f44983e1fbbb\" (UID: \"e1624891-a22f-4ed4-8590-f44983e1fbbb\") "
Nov 02 15:00:43.139005 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.138902    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-util\" (UniqueName: \"kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-util\") pod \"1c5c0540-2328-49c6-b198-57d221d88c0a\" (UID: \"1c5c0540-2328-49c6-b198-57d221d88c0a\") "
Nov 02 15:00:43.139005 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.138938    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1624891-a22f-4ed4-8590-f44983e1fbbb-kube-api-access-sskkd\" (UniqueName: \"kubernetes.io/projected/e1624891-a22f-4ed4-8590-f44983e1fbbb-kube-api-access-sskkd\") pod \"e1624891-a22f-4ed4-8590-f44983e1fbbb\" (UID: \"e1624891-a22f-4ed4-8590-f44983e1fbbb\") "
Nov 02 15:00:43.139005 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.138975    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5c0540-2328-49c6-b198-57d221d88c0a-kube-api-access-92pbh\" (UniqueName: \"kubernetes.io/projected/1c5c0540-2328-49c6-b198-57d221d88c0a-kube-api-access-92pbh\") pod \"1c5c0540-2328-49c6-b198-57d221d88c0a\" (UID: \"1c5c0540-2328-49c6-b198-57d221d88c0a\") "
Nov 02 15:00:43.139145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.139007    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-bundle\") pod \"e1624891-a22f-4ed4-8590-f44983e1fbbb\" (UID: \"e1624891-a22f-4ed4-8590-f44983e1fbbb\") "
Nov 02 15:00:43.139145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.139075    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-bundle\") pod \"1c5c0540-2328-49c6-b198-57d221d88c0a\" (UID: \"1c5c0540-2328-49c6-b198-57d221d88c0a\") "
Nov 02 15:00:43.139553 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.139527    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-bundle" pod "e1624891-a22f-4ed4-8590-f44983e1fbbb" (UID: "e1624891-a22f-4ed4-8590-f44983e1fbbb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:43.139757 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.139730    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-bundle" pod "1c5c0540-2328-49c6-b198-57d221d88c0a" (UID: "1c5c0540-2328-49c6-b198-57d221d88c0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:43.141111 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.141085    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5c0540-2328-49c6-b198-57d221d88c0a-kube-api-access-92pbh" pod "1c5c0540-2328-49c6-b198-57d221d88c0a" (UID: "1c5c0540-2328-49c6-b198-57d221d88c0a"). InnerVolumeSpecName "kube-api-access-92pbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:43.141454 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.141426    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1624891-a22f-4ed4-8590-f44983e1fbbb-kube-api-access-sskkd" pod "e1624891-a22f-4ed4-8590-f44983e1fbbb" (UID: "e1624891-a22f-4ed4-8590-f44983e1fbbb"). InnerVolumeSpecName "kube-api-access-sskkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:43.145103 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.145081    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-util" pod "1c5c0540-2328-49c6-b198-57d221d88c0a" (UID: "1c5c0540-2328-49c6-b198-57d221d88c0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:43.145926 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.145764    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-util" pod "e1624891-a22f-4ed4-8590-f44983e1fbbb" (UID: "e1624891-a22f-4ed4-8590-f44983e1fbbb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:43.240010 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.239930    4891 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-util\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:43.240010 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.239959    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sskkd\" (UniqueName: \"kubernetes.io/projected/e1624891-a22f-4ed4-8590-f44983e1fbbb-kube-api-access-sskkd\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:43.240010 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.239969    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92pbh\" (UniqueName: \"kubernetes.io/projected/1c5c0540-2328-49c6-b198-57d221d88c0a-kube-api-access-92pbh\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:43.240010 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.239978    4891 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:43.240010 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.239987    4891 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c5c0540-2328-49c6-b198-57d221d88c0a-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:43.240010 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.239995    4891 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1624891-a22f-4ed4-8590-f44983e1fbbb-util\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:43.873261 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.873234    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr"
Nov 02 15:00:43.873261 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.873236    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pw4cr" event={"ID":"e1624891-a22f-4ed4-8590-f44983e1fbbb","Type":"ContainerDied","Data":"8989bad80181b6d9f83f89a957876f7652f3737baaad473976c6900013a9e5c0"}
Nov 02 15:00:43.873780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.873299    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8989bad80181b6d9f83f89a957876f7652f3737baaad473976c6900013a9e5c0"
Nov 02 15:00:43.875281 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.875254    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22"
Nov 02 15:00:43.875414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.875274    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwhm22" event={"ID":"1c5c0540-2328-49c6-b198-57d221d88c0a","Type":"ContainerDied","Data":"00702aeed707606f538a542d290a36b4e4327683ac5ce668ae3538d6619eab27"}
Nov 02 15:00:43.875414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.875331    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00702aeed707606f538a542d290a36b4e4327683ac5ce668ae3538d6619eab27"
Nov 02 15:00:43.877554 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.877527    4891 generic.go:356] "Generic (PLEG): container finished" podID="f3b70660-5510-4dd2-9e89-83cc2193fc93" containerID="9a21264cddc807036d2f257c41a7249a38af222418d0096884f5506191b297aa" exitCode=0
Nov 02 15:00:43.877675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:43.877583    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x" event={"ID":"f3b70660-5510-4dd2-9e89-83cc2193fc93","Type":"ContainerDied","Data":"9a21264cddc807036d2f257c41a7249a38af222418d0096884f5506191b297aa"}
Nov 02 15:00:44.271938 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:00:44.271906    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:00:45.012285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.012253    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:45.153132 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.153047    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-bundle\") pod \"f3b70660-5510-4dd2-9e89-83cc2193fc93\" (UID: \"f3b70660-5510-4dd2-9e89-83cc2193fc93\") "
Nov 02 15:00:45.153132 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.153118    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-util\" (UniqueName: \"kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-util\") pod \"f3b70660-5510-4dd2-9e89-83cc2193fc93\" (UID: \"f3b70660-5510-4dd2-9e89-83cc2193fc93\") "
Nov 02 15:00:45.153383 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.153138    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b70660-5510-4dd2-9e89-83cc2193fc93-kube-api-access-x7hjr\" (UniqueName: \"kubernetes.io/projected/f3b70660-5510-4dd2-9e89-83cc2193fc93-kube-api-access-x7hjr\") pod \"f3b70660-5510-4dd2-9e89-83cc2193fc93\" (UID: \"f3b70660-5510-4dd2-9e89-83cc2193fc93\") "
Nov 02 15:00:45.153532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.153508    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-bundle" pod "f3b70660-5510-4dd2-9e89-83cc2193fc93" (UID: "f3b70660-5510-4dd2-9e89-83cc2193fc93"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:45.155221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.155195    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b70660-5510-4dd2-9e89-83cc2193fc93-kube-api-access-x7hjr" pod "f3b70660-5510-4dd2-9e89-83cc2193fc93" (UID: "f3b70660-5510-4dd2-9e89-83cc2193fc93"). InnerVolumeSpecName "kube-api-access-x7hjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:45.158335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.158301    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-util" pod "f3b70660-5510-4dd2-9e89-83cc2193fc93" (UID: "f3b70660-5510-4dd2-9e89-83cc2193fc93"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:45.253914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.253871    4891 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-util\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:45.253914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.253907    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7hjr\" (UniqueName: \"kubernetes.io/projected/f3b70660-5510-4dd2-9e89-83cc2193fc93-kube-api-access-x7hjr\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:45.253914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.253917    4891 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3b70660-5510-4dd2-9e89-83cc2193fc93-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:45.888495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.888469    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x"
Nov 02 15:00:45.888638 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.888469    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vwz2x" event={"ID":"f3b70660-5510-4dd2-9e89-83cc2193fc93","Type":"ContainerDied","Data":"5d8c16a3d1bb1f0fce288f7938d6bce25c9de16e964960b839be1dbe68ea95d0"}
Nov 02 15:00:45.888638 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:45.888585    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d8c16a3d1bb1f0fce288f7938d6bce25c9de16e964960b839be1dbe68ea95d0"
Nov 02 15:00:48.890695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.890663    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7"]
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.890949    4891 cpu_manager.go:414] "containerMap: removing container" podUID="c0b8c1b2-8f94-40f0-a758-f1027a797b69" containerName="util"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.890959    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b8c1b2-8f94-40f0-a758-f1027a797b69" containerName="util"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.890970    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1c5c0540-2328-49c6-b198-57d221d88c0a" containerName="extract"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.890975    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5c0540-2328-49c6-b198-57d221d88c0a" containerName="extract"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.890983    4891 cpu_manager.go:414] "containerMap: removing container" podUID="e1cb9fa4-4a58-459d-9bfc-09d650cf3657" containerName="pull"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.890990    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1cb9fa4-4a58-459d-9bfc-09d650cf3657" containerName="pull"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.890998    4891 cpu_manager.go:414] "containerMap: removing container" podUID="6655aa2e-f88a-45f2-a5d7-927a9802ab4c" containerName="pull"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891003    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6655aa2e-f88a-45f2-a5d7-927a9802ab4c" containerName="pull"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891010    4891 cpu_manager.go:414] "containerMap: removing container" podUID="b0086234-e43e-40f0-be10-217afb3ba74d" containerName="pull"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891014    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0086234-e43e-40f0-be10-217afb3ba74d" containerName="pull"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891021    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" containerName="util"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891026    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" containerName="util"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891033    4891 cpu_manager.go:414] "containerMap: removing container" podUID="6655aa2e-f88a-45f2-a5d7-927a9802ab4c" containerName="util"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891037    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6655aa2e-f88a-45f2-a5d7-927a9802ab4c" containerName="util"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891043    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" containerName="pull"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891047    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" containerName="pull"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891051    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1c5c0540-2328-49c6-b198-57d221d88c0a" containerName="pull"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891055    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5c0540-2328-49c6-b198-57d221d88c0a" containerName="pull"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891061    4891 cpu_manager.go:414] "containerMap: removing container" podUID="6655aa2e-f88a-45f2-a5d7-927a9802ab4c" containerName="extract"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891065    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6655aa2e-f88a-45f2-a5d7-927a9802ab4c" containerName="extract"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891073    4891 cpu_manager.go:414] "containerMap: removing container" podUID="e1624891-a22f-4ed4-8590-f44983e1fbbb" containerName="extract"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891077    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1624891-a22f-4ed4-8590-f44983e1fbbb" containerName="extract"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891085    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1f934255-4e4b-4d05-91a7-15ce08f174e4" containerName="util"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891089    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f934255-4e4b-4d05-91a7-15ce08f174e4" containerName="util"
Nov 02 15:00:48.891080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891094    4891 cpu_manager.go:414] "containerMap: removing container" podUID="e1cb9fa4-4a58-459d-9bfc-09d650cf3657" containerName="util"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891099    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1cb9fa4-4a58-459d-9bfc-09d650cf3657" containerName="util"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891111    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f3b70660-5510-4dd2-9e89-83cc2193fc93" containerName="util"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891115    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b70660-5510-4dd2-9e89-83cc2193fc93" containerName="util"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891123    4891 cpu_manager.go:414] "containerMap: removing container" podUID="e1624891-a22f-4ed4-8590-f44983e1fbbb" containerName="pull"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891128    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1624891-a22f-4ed4-8590-f44983e1fbbb" containerName="pull"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891133    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f3b70660-5510-4dd2-9e89-83cc2193fc93" containerName="pull"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891137    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b70660-5510-4dd2-9e89-83cc2193fc93" containerName="pull"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891143    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1f934255-4e4b-4d05-91a7-15ce08f174e4" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891147    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f934255-4e4b-4d05-91a7-15ce08f174e4" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891154    4891 cpu_manager.go:414] "containerMap: removing container" podUID="e1cb9fa4-4a58-459d-9bfc-09d650cf3657" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891158    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1cb9fa4-4a58-459d-9bfc-09d650cf3657" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891164    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1c5c0540-2328-49c6-b198-57d221d88c0a" containerName="util"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891168    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5c0540-2328-49c6-b198-57d221d88c0a" containerName="util"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891173    4891 cpu_manager.go:414] "containerMap: removing container" podUID="b0086234-e43e-40f0-be10-217afb3ba74d" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891177    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0086234-e43e-40f0-be10-217afb3ba74d" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891182    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891186    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891190    4891 cpu_manager.go:414] "containerMap: removing container" podUID="c0b8c1b2-8f94-40f0-a758-f1027a797b69" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891194    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b8c1b2-8f94-40f0-a758-f1027a797b69" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891199    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f3b70660-5510-4dd2-9e89-83cc2193fc93" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891203    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b70660-5510-4dd2-9e89-83cc2193fc93" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891211    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1f934255-4e4b-4d05-91a7-15ce08f174e4" containerName="pull"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891215    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f934255-4e4b-4d05-91a7-15ce08f174e4" containerName="pull"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891219    4891 cpu_manager.go:414] "containerMap: removing container" podUID="b0086234-e43e-40f0-be10-217afb3ba74d" containerName="util"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891223    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0086234-e43e-40f0-be10-217afb3ba74d" containerName="util"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891229    4891 cpu_manager.go:414] "containerMap: removing container" podUID="e1624891-a22f-4ed4-8590-f44983e1fbbb" containerName="util"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891232    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1624891-a22f-4ed4-8590-f44983e1fbbb" containerName="util"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891238    4891 cpu_manager.go:414] "containerMap: removing container" podUID="c0b8c1b2-8f94-40f0-a758-f1027a797b69" containerName="pull"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891242    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b8c1b2-8f94-40f0-a758-f1027a797b69" containerName="pull"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891290    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5c0540-2328-49c6-b198-57d221d88c0a" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891298    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="6655aa2e-f88a-45f2-a5d7-927a9802ab4c" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891304    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1624891-a22f-4ed4-8590-f44983e1fbbb" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891323    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f934255-4e4b-4d05-91a7-15ce08f174e4" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891329    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b70660-5510-4dd2-9e89-83cc2193fc93" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891334    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1cb9fa4-4a58-459d-9bfc-09d650cf3657" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891339    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0086234-e43e-40f0-be10-217afb3ba74d" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891344    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf1bed8-3a81-47af-ad2a-d401fc22d0dc" containerName="extract"
Nov 02 15:00:48.891775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.891349    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b8c1b2-8f94-40f0-a758-f1027a797b69" containerName="extract"
Nov 02 15:00:48.893211 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.893194    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7"
Nov 02 15:00:48.983081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.983048    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2k6h\" (UniqueName: \"kubernetes.io/projected/f9d52ccf-8779-4666-b4eb-8ee214205d18-kube-api-access-b2k6h\") pod \"jobset-operator-674cd59dc5-pflz7\" (UID: \"f9d52ccf-8779-4666-b4eb-8ee214205d18\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7"
Nov 02 15:00:48.983241 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.983119    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d52ccf-8779-4666-b4eb-8ee214205d18-tmp\") pod \"jobset-operator-674cd59dc5-pflz7\" (UID: \"f9d52ccf-8779-4666-b4eb-8ee214205d18\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7"
Nov 02 15:00:48.996689 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.996659    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-5vfgb\""
Nov 02 15:00:48.996834 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.996710    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 15:00:48.996834 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.996751    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 15:00:48.998431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:48.998410    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7"]
Nov 02 15:00:49.050305 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.050271    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"]
Nov 02 15:00:49.052794 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.052778    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:49.084381 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.084348    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d52ccf-8779-4666-b4eb-8ee214205d18-tmp\") pod \"jobset-operator-674cd59dc5-pflz7\" (UID: \"f9d52ccf-8779-4666-b4eb-8ee214205d18\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7"
Nov 02 15:00:49.084542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.084409    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2k6h\" (UniqueName: \"kubernetes.io/projected/f9d52ccf-8779-4666-b4eb-8ee214205d18-kube-api-access-b2k6h\") pod \"jobset-operator-674cd59dc5-pflz7\" (UID: \"f9d52ccf-8779-4666-b4eb-8ee214205d18\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7"
Nov 02 15:00:49.084715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.084696    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d52ccf-8779-4666-b4eb-8ee214205d18-tmp\") pod \"jobset-operator-674cd59dc5-pflz7\" (UID: \"f9d52ccf-8779-4666-b4eb-8ee214205d18\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7"
Nov 02 15:00:49.105599 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.105576    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 15:00:49.105686 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.105652    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 15:00:49.106463 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.106449    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7rb6r\""
Nov 02 15:00:49.114159 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.114131    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"]
Nov 02 15:00:49.144913 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.144850    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2k6h\" (UniqueName: \"kubernetes.io/projected/f9d52ccf-8779-4666-b4eb-8ee214205d18-kube-api-access-b2k6h\") pod \"jobset-operator-674cd59dc5-pflz7\" (UID: \"f9d52ccf-8779-4666-b4eb-8ee214205d18\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7"
Nov 02 15:00:49.184972 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.184935    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxknn\" (UniqueName: \"kubernetes.io/projected/a3147f52-2965-4186-bed8-d93793c48581-kube-api-access-rxknn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89\" (UID: \"a3147f52-2965-4186-bed8-d93793c48581\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:49.185119 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.184994    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89\" (UID: \"a3147f52-2965-4186-bed8-d93793c48581\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:49.185179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.185126    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89\" (UID: \"a3147f52-2965-4186-bed8-d93793c48581\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:49.222036 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.222003    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7"
Nov 02 15:00:49.285906 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.285874    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89\" (UID: \"a3147f52-2965-4186-bed8-d93793c48581\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:49.286056 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.285935    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxknn\" (UniqueName: \"kubernetes.io/projected/a3147f52-2965-4186-bed8-d93793c48581-kube-api-access-rxknn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89\" (UID: \"a3147f52-2965-4186-bed8-d93793c48581\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:49.286056 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.286005    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89\" (UID: \"a3147f52-2965-4186-bed8-d93793c48581\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:49.286369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.286346    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89\" (UID: \"a3147f52-2965-4186-bed8-d93793c48581\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:49.286547 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.286529    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89\" (UID: \"a3147f52-2965-4186-bed8-d93793c48581\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:49.346706 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.346672    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxknn\" (UniqueName: \"kubernetes.io/projected/a3147f52-2965-4186-bed8-d93793c48581-kube-api-access-rxknn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89\" (UID: \"a3147f52-2965-4186-bed8-d93793c48581\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:49.365488 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.365458    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:49.600229 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.600204    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7"]
Nov 02 15:00:49.602749 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:49.602720    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9d52ccf_8779_4666_b4eb_8ee214205d18.slice/crio-12116f64caf27ebcecffe3ce4d7a30795516f58463ceb86baf399bd2df382ce2 WatchSource:0}: Error finding container 12116f64caf27ebcecffe3ce4d7a30795516f58463ceb86baf399bd2df382ce2: Status 404 returned error can't find the container with id 12116f64caf27ebcecffe3ce4d7a30795516f58463ceb86baf399bd2df382ce2
Nov 02 15:00:49.736072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.736048    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"]
Nov 02 15:00:49.737934 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:49.737899    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3147f52_2965_4186_bed8_d93793c48581.slice/crio-fc2e45aca6772287e8299d05797237bd334ca4a53294bffd230f98407e0423f0 WatchSource:0}: Error finding container fc2e45aca6772287e8299d05797237bd334ca4a53294bffd230f98407e0423f0: Status 404 returned error can't find the container with id fc2e45aca6772287e8299d05797237bd334ca4a53294bffd230f98407e0423f0
Nov 02 15:00:49.908292 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.908206    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7" event={"ID":"f9d52ccf-8779-4666-b4eb-8ee214205d18","Type":"ContainerStarted","Data":"12116f64caf27ebcecffe3ce4d7a30795516f58463ceb86baf399bd2df382ce2"}
Nov 02 15:00:49.910165 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.910134    4891 generic.go:356] "Generic (PLEG): container finished" podID="a3147f52-2965-4186-bed8-d93793c48581" containerID="cc86ecee9878dd8ccc7ebcff5a10d619df4f3a5f80efa44d7617b35a9dc6281d" exitCode=0
Nov 02 15:00:49.910253 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.910180    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89" event={"ID":"a3147f52-2965-4186-bed8-d93793c48581","Type":"ContainerDied","Data":"cc86ecee9878dd8ccc7ebcff5a10d619df4f3a5f80efa44d7617b35a9dc6281d"}
Nov 02 15:00:49.910253 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:49.910215    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89" event={"ID":"a3147f52-2965-4186-bed8-d93793c48581","Type":"ContainerStarted","Data":"fc2e45aca6772287e8299d05797237bd334ca4a53294bffd230f98407e0423f0"}
Nov 02 15:00:50.060081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.060046    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"]
Nov 02 15:00:50.062372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.062356    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:50.114461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.114433    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2gsxf\""
Nov 02 15:00:50.166044 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.166015    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"]
Nov 02 15:00:50.194098 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.194056    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42\" (UID: \"ee0259a2-5692-4d8c-b799-f8598b628250\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:50.194264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.194107    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzl46\" (UniqueName: \"kubernetes.io/projected/ee0259a2-5692-4d8c-b799-f8598b628250-kube-api-access-tzl46\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42\" (UID: \"ee0259a2-5692-4d8c-b799-f8598b628250\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:50.194264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.194238    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42\" (UID: \"ee0259a2-5692-4d8c-b799-f8598b628250\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:50.295200 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.295161    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzl46\" (UniqueName: \"kubernetes.io/projected/ee0259a2-5692-4d8c-b799-f8598b628250-kube-api-access-tzl46\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42\" (UID: \"ee0259a2-5692-4d8c-b799-f8598b628250\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:50.295404 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.295241    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42\" (UID: \"ee0259a2-5692-4d8c-b799-f8598b628250\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:50.295404 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.295265    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42\" (UID: \"ee0259a2-5692-4d8c-b799-f8598b628250\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:50.295696 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.295676    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42\" (UID: \"ee0259a2-5692-4d8c-b799-f8598b628250\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:50.295740 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.295693    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42\" (UID: \"ee0259a2-5692-4d8c-b799-f8598b628250\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:50.356574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.356497    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzl46\" (UniqueName: \"kubernetes.io/projected/ee0259a2-5692-4d8c-b799-f8598b628250-kube-api-access-tzl46\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42\" (UID: \"ee0259a2-5692-4d8c-b799-f8598b628250\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:50.374342 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.374303    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:50.751118 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.751088    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"]
Nov 02 15:00:50.753456 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:50.753426    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0259a2_5692_4d8c_b799_f8598b628250.slice/crio-2c851f3dcfc52aaa43a1b9ab9fa0e0379f60cd7845237bf72709e3590117f7e7 WatchSource:0}: Error finding container 2c851f3dcfc52aaa43a1b9ab9fa0e0379f60cd7845237bf72709e3590117f7e7: Status 404 returned error can't find the container with id 2c851f3dcfc52aaa43a1b9ab9fa0e0379f60cd7845237bf72709e3590117f7e7
Nov 02 15:00:50.916359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.916267    4891 generic.go:356] "Generic (PLEG): container finished" podID="ee0259a2-5692-4d8c-b799-f8598b628250" containerID="3c059d373d275c251eaa2beea1dfb68cf53d57853191a0403cd04e9eb62e53ab" exitCode=0
Nov 02 15:00:50.916767 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.916350    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42" event={"ID":"ee0259a2-5692-4d8c-b799-f8598b628250","Type":"ContainerDied","Data":"3c059d373d275c251eaa2beea1dfb68cf53d57853191a0403cd04e9eb62e53ab"}
Nov 02 15:00:50.916767 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.916408    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42" event={"ID":"ee0259a2-5692-4d8c-b799-f8598b628250","Type":"ContainerStarted","Data":"2c851f3dcfc52aaa43a1b9ab9fa0e0379f60cd7845237bf72709e3590117f7e7"}
Nov 02 15:00:50.918458 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.918434    4891 generic.go:356] "Generic (PLEG): container finished" podID="a3147f52-2965-4186-bed8-d93793c48581" containerID="d4226ac5474e95d09fa2a20a6ec949b4f2015b290707230f09dc7eec20d03e36" exitCode=0
Nov 02 15:00:50.918556 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:50.918473    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89" event={"ID":"a3147f52-2965-4186-bed8-d93793c48581","Type":"ContainerDied","Data":"d4226ac5474e95d09fa2a20a6ec949b4f2015b290707230f09dc7eec20d03e36"}
Nov 02 15:00:51.928616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:51.928537    4891 generic.go:356] "Generic (PLEG): container finished" podID="ee0259a2-5692-4d8c-b799-f8598b628250" containerID="08571951ae95cd6c7a1536e5429def10152f6cf7728ded2d4160f6f02738f536" exitCode=0
Nov 02 15:00:51.929039 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:51.928628    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42" event={"ID":"ee0259a2-5692-4d8c-b799-f8598b628250","Type":"ContainerDied","Data":"08571951ae95cd6c7a1536e5429def10152f6cf7728ded2d4160f6f02738f536"}
Nov 02 15:00:51.931436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:51.931409    4891 generic.go:356] "Generic (PLEG): container finished" podID="a3147f52-2965-4186-bed8-d93793c48581" containerID="57bbeee34662d629b33b570002640ec87bbffbe8d739ca0a40d194c9abb6b6fb" exitCode=0
Nov 02 15:00:51.931548 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:51.931463    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89" event={"ID":"a3147f52-2965-4186-bed8-d93793c48581","Type":"ContainerDied","Data":"57bbeee34662d629b33b570002640ec87bbffbe8d739ca0a40d194c9abb6b6fb"}
Nov 02 15:00:52.939021 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:52.938939    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7" event={"ID":"f9d52ccf-8779-4666-b4eb-8ee214205d18","Type":"ContainerStarted","Data":"20f3cbbdd3ace7e7815c5e1e09f1de267024a2e8364ff1320a0a33754c400e70"}
Nov 02 15:00:52.941443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:52.941416    4891 generic.go:356] "Generic (PLEG): container finished" podID="ee0259a2-5692-4d8c-b799-f8598b628250" containerID="d4665081b6ceb350acadba3a82fd3260cf036cce64db3f90c6a862464352ebb5" exitCode=0
Nov 02 15:00:52.941573 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:52.941481    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42" event={"ID":"ee0259a2-5692-4d8c-b799-f8598b628250","Type":"ContainerDied","Data":"d4665081b6ceb350acadba3a82fd3260cf036cce64db3f90c6a862464352ebb5"}
Nov 02 15:00:53.062908 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.062863    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pflz7" podStartSLOduration=2.102685358 podStartE2EDuration="5.062848008s" totalImagesPullingTime="2.96016265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:53.061407651 +0000 UTC m=+260.654024451" watchObservedRunningTime="2026-11-02 15:00:53.062848008 +0000 UTC m=+260.655464807"
Nov 02 15:00:53.064081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.064064    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:53.228676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.228580    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3147f52-2965-4186-bed8-d93793c48581-kube-api-access-rxknn\" (UniqueName: \"kubernetes.io/projected/a3147f52-2965-4186-bed8-d93793c48581-kube-api-access-rxknn\") pod \"a3147f52-2965-4186-bed8-d93793c48581\" (UID: \"a3147f52-2965-4186-bed8-d93793c48581\") "
Nov 02 15:00:53.228676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.228622    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-bundle\") pod \"a3147f52-2965-4186-bed8-d93793c48581\" (UID: \"a3147f52-2965-4186-bed8-d93793c48581\") "
Nov 02 15:00:53.228676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.228649    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-util\" (UniqueName: \"kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-util\") pod \"a3147f52-2965-4186-bed8-d93793c48581\" (UID: \"a3147f52-2965-4186-bed8-d93793c48581\") "
Nov 02 15:00:53.229260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.229222    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-bundle" pod "a3147f52-2965-4186-bed8-d93793c48581" (UID: "a3147f52-2965-4186-bed8-d93793c48581"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:53.230747 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.230727    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3147f52-2965-4186-bed8-d93793c48581-kube-api-access-rxknn" pod "a3147f52-2965-4186-bed8-d93793c48581" (UID: "a3147f52-2965-4186-bed8-d93793c48581"). InnerVolumeSpecName "kube-api-access-rxknn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:53.234682 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.234640    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-util" pod "a3147f52-2965-4186-bed8-d93793c48581" (UID: "a3147f52-2965-4186-bed8-d93793c48581"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:53.329358 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.329319    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxknn\" (UniqueName: \"kubernetes.io/projected/a3147f52-2965-4186-bed8-d93793c48581-kube-api-access-rxknn\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:53.329358 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.329360    4891 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:53.329567 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.329376    4891 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3147f52-2965-4186-bed8-d93793c48581-util\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:53.948969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.948939    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89"
Nov 02 15:00:53.948969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.948960    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vwd89" event={"ID":"a3147f52-2965-4186-bed8-d93793c48581","Type":"ContainerDied","Data":"fc2e45aca6772287e8299d05797237bd334ca4a53294bffd230f98407e0423f0"}
Nov 02 15:00:53.949530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:53.949002    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc2e45aca6772287e8299d05797237bd334ca4a53294bffd230f98407e0423f0"
Nov 02 15:00:54.072922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.072892    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:54.237215 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.237121    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee0259a2-5692-4d8c-b799-f8598b628250-kube-api-access-tzl46\" (UniqueName: \"kubernetes.io/projected/ee0259a2-5692-4d8c-b799-f8598b628250-kube-api-access-tzl46\") pod \"ee0259a2-5692-4d8c-b799-f8598b628250\" (UID: \"ee0259a2-5692-4d8c-b799-f8598b628250\") "
Nov 02 15:00:54.237215 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.237198    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-util\" (UniqueName: \"kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-util\") pod \"ee0259a2-5692-4d8c-b799-f8598b628250\" (UID: \"ee0259a2-5692-4d8c-b799-f8598b628250\") "
Nov 02 15:00:54.237215 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.237219    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-bundle\") pod \"ee0259a2-5692-4d8c-b799-f8598b628250\" (UID: \"ee0259a2-5692-4d8c-b799-f8598b628250\") "
Nov 02 15:00:54.238035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.238001    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-bundle" pod "ee0259a2-5692-4d8c-b799-f8598b628250" (UID: "ee0259a2-5692-4d8c-b799-f8598b628250"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:54.239295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.239272    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0259a2-5692-4d8c-b799-f8598b628250-kube-api-access-tzl46" pod "ee0259a2-5692-4d8c-b799-f8598b628250" (UID: "ee0259a2-5692-4d8c-b799-f8598b628250"). InnerVolumeSpecName "kube-api-access-tzl46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:54.243955 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.243913    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-util" pod "ee0259a2-5692-4d8c-b799-f8598b628250" (UID: "ee0259a2-5692-4d8c-b799-f8598b628250"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:54.338289 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.338253    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzl46\" (UniqueName: \"kubernetes.io/projected/ee0259a2-5692-4d8c-b799-f8598b628250-kube-api-access-tzl46\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:54.338289 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.338281    4891 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-util\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:54.338289 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.338292    4891 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee0259a2-5692-4d8c-b799-f8598b628250-bundle\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:00:54.960277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.960177    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42" event={"ID":"ee0259a2-5692-4d8c-b799-f8598b628250","Type":"ContainerDied","Data":"2c851f3dcfc52aaa43a1b9ab9fa0e0379f60cd7845237bf72709e3590117f7e7"}
Nov 02 15:00:54.960277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.960211    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqzn42"
Nov 02 15:00:54.960277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:54.960265    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c851f3dcfc52aaa43a1b9ab9fa0e0379f60cd7845237bf72709e3590117f7e7"
Nov 02 15:00:58.008063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008018    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2"]
Nov 02 15:00:58.008609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008496    4891 cpu_manager.go:414] "containerMap: removing container" podUID="a3147f52-2965-4186-bed8-d93793c48581" containerName="extract"
Nov 02 15:00:58.008609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008514    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3147f52-2965-4186-bed8-d93793c48581" containerName="extract"
Nov 02 15:00:58.008609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008544    4891 cpu_manager.go:414] "containerMap: removing container" podUID="a3147f52-2965-4186-bed8-d93793c48581" containerName="pull"
Nov 02 15:00:58.008609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008551    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3147f52-2965-4186-bed8-d93793c48581" containerName="pull"
Nov 02 15:00:58.008609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008561    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ee0259a2-5692-4d8c-b799-f8598b628250" containerName="util"
Nov 02 15:00:58.008609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008568    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0259a2-5692-4d8c-b799-f8598b628250" containerName="util"
Nov 02 15:00:58.008609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008582    4891 cpu_manager.go:414] "containerMap: removing container" podUID="a3147f52-2965-4186-bed8-d93793c48581" containerName="util"
Nov 02 15:00:58.008609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008589    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3147f52-2965-4186-bed8-d93793c48581" containerName="util"
Nov 02 15:00:58.008609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008598    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ee0259a2-5692-4d8c-b799-f8598b628250" containerName="pull"
Nov 02 15:00:58.008609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008604    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0259a2-5692-4d8c-b799-f8598b628250" containerName="pull"
Nov 02 15:00:58.009080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008635    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ee0259a2-5692-4d8c-b799-f8598b628250" containerName="extract"
Nov 02 15:00:58.009080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008642    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0259a2-5692-4d8c-b799-f8598b628250" containerName="extract"
Nov 02 15:00:58.009080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008741    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee0259a2-5692-4d8c-b799-f8598b628250" containerName="extract"
Nov 02 15:00:58.009080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.008756    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3147f52-2965-4186-bed8-d93793c48581" containerName="extract"
Nov 02 15:00:58.010980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.010958    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2"
Nov 02 15:00:58.078170 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.078139    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 15:00:58.127773 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.127744    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 15:00:58.127773 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.127744    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gmxmh\""
Nov 02 15:00:58.129960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.129935    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2"]
Nov 02 15:00:58.170006 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.169971    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdblw\" (UniqueName: \"kubernetes.io/projected/12ce1ac0-a29b-4dde-867d-1ae8e712ff41-kube-api-access-wdblw\") pod \"authorino-operator-7b7bb4bc77-hr5x2\" (UID: \"12ce1ac0-a29b-4dde-867d-1ae8e712ff41\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2"
Nov 02 15:00:58.270954 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.270927    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdblw\" (UniqueName: \"kubernetes.io/projected/12ce1ac0-a29b-4dde-867d-1ae8e712ff41-kube-api-access-wdblw\") pod \"authorino-operator-7b7bb4bc77-hr5x2\" (UID: \"12ce1ac0-a29b-4dde-867d-1ae8e712ff41\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2"
Nov 02 15:00:58.333288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.333252    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdblw\" (UniqueName: \"kubernetes.io/projected/12ce1ac0-a29b-4dde-867d-1ae8e712ff41-kube-api-access-wdblw\") pod \"authorino-operator-7b7bb4bc77-hr5x2\" (UID: \"12ce1ac0-a29b-4dde-867d-1ae8e712ff41\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2"
Nov 02 15:00:58.622560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.622479    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2"
Nov 02 15:00:58.994890 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:58.994781    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2"]
Nov 02 15:00:58.997683 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:00:58.997651    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ce1ac0_a29b_4dde_867d_1ae8e712ff41.slice/crio-6acebac225328bd0a2f3bd105a4f9d6aa347ce963caad8bc2e2684ac94a0fc33 WatchSource:0}: Error finding container 6acebac225328bd0a2f3bd105a4f9d6aa347ce963caad8bc2e2684ac94a0fc33: Status 404 returned error can't find the container with id 6acebac225328bd0a2f3bd105a4f9d6aa347ce963caad8bc2e2684ac94a0fc33
Nov 02 15:00:59.984813 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:00:59.984778    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2" event={"ID":"12ce1ac0-a29b-4dde-867d-1ae8e712ff41","Type":"ContainerStarted","Data":"6acebac225328bd0a2f3bd105a4f9d6aa347ce963caad8bc2e2684ac94a0fc33"}
Nov 02 15:01:01.733818 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:01.733785    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd"]
Nov 02 15:01:01.742661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:01.742632    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd"
Nov 02 15:01:01.799058 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:01.799012    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 15:01:01.848087 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:01.848051    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-67x44\""
Nov 02 15:01:01.850432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:01.850399    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd"]
Nov 02 15:01:01.898072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:01.898036    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtbjb\" (UniqueName: \"kubernetes.io/projected/8bdb971a-ff5a-4c55-aa72-0de03bb0a4f8-kube-api-access-qtbjb\") pod \"dns-operator-controller-manager-8557cbdb84-9xdqd\" (UID: \"8bdb971a-ff5a-4c55-aa72-0de03bb0a4f8\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd"
Nov 02 15:01:01.998905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:01.998830    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtbjb\" (UniqueName: \"kubernetes.io/projected/8bdb971a-ff5a-4c55-aa72-0de03bb0a4f8-kube-api-access-qtbjb\") pod \"dns-operator-controller-manager-8557cbdb84-9xdqd\" (UID: \"8bdb971a-ff5a-4c55-aa72-0de03bb0a4f8\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd"
Nov 02 15:01:02.078406 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:02.078369    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtbjb\" (UniqueName: \"kubernetes.io/projected/8bdb971a-ff5a-4c55-aa72-0de03bb0a4f8-kube-api-access-qtbjb\") pod \"dns-operator-controller-manager-8557cbdb84-9xdqd\" (UID: \"8bdb971a-ff5a-4c55-aa72-0de03bb0a4f8\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd"
Nov 02 15:01:02.357280 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:02.357241    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd"
Nov 02 15:01:02.747261 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:02.747180    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd"]
Nov 02 15:01:02.750784 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:01:02.750755    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdb971a_ff5a_4c55_aa72_0de03bb0a4f8.slice/crio-d0413a861e9a1f7e4617a1da3b03a3a873fb2cad9cc9368a7be2a1223a8211a6 WatchSource:0}: Error finding container d0413a861e9a1f7e4617a1da3b03a3a873fb2cad9cc9368a7be2a1223a8211a6: Status 404 returned error can't find the container with id d0413a861e9a1f7e4617a1da3b03a3a873fb2cad9cc9368a7be2a1223a8211a6
Nov 02 15:01:03.006962 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.006855    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd" event={"ID":"8bdb971a-ff5a-4c55-aa72-0de03bb0a4f8","Type":"ContainerStarted","Data":"d0413a861e9a1f7e4617a1da3b03a3a873fb2cad9cc9368a7be2a1223a8211a6"}
Nov 02 15:01:03.008583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.008549    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2" event={"ID":"12ce1ac0-a29b-4dde-867d-1ae8e712ff41","Type":"ContainerStarted","Data":"7f9a2bfe7f37dcddb510542bbc1a7a23a6249fdfd67d7b2b52ac2bbd04199699"}
Nov 02 15:01:03.008734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.008716    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2"
Nov 02 15:01:03.127778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.127725    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2" podStartSLOduration=2.626881571 podStartE2EDuration="6.127710288s" totalImagesPullingTime="3.500828717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:03.126642366 +0000 UTC m=+270.719259165" watchObservedRunningTime="2026-11-02 15:01:03.127710288 +0000 UTC m=+270.720327088"
Nov 02 15:01:03.400739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.400701    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"]
Nov 02 15:01:03.404488 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.404472    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.408919 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.408893    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad02b257-14a7-415f-835e-2bb332e7e1f1-tmp\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.409034 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.408932    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad02b257-14a7-415f-835e-2bb332e7e1f1-manifests\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.409034 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.408949    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad02b257-14a7-415f-835e-2bb332e7e1f1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.409110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.409049    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad02b257-14a7-415f-835e-2bb332e7e1f1-metrics-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.409144 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.409116    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl7s2\" (UniqueName: \"kubernetes.io/projected/ad02b257-14a7-415f-835e-2bb332e7e1f1-kube-api-access-sl7s2\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.409177 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.409144    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad02b257-14a7-415f-835e-2bb332e7e1f1-webhook-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.481095 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.481054    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 15:01:03.481095 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.481092    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 15:01:03.481341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.481104    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 15:01:03.481341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.481169    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 15:01:03.481341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.481225    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-5tklz\""
Nov 02 15:01:03.509977 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.509940    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad02b257-14a7-415f-835e-2bb332e7e1f1-manifests\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.510147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.509987    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad02b257-14a7-415f-835e-2bb332e7e1f1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.510147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.510024    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad02b257-14a7-415f-835e-2bb332e7e1f1-metrics-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.510147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.510080    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl7s2\" (UniqueName: \"kubernetes.io/projected/ad02b257-14a7-415f-835e-2bb332e7e1f1-kube-api-access-sl7s2\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.510147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.510115    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad02b257-14a7-415f-835e-2bb332e7e1f1-webhook-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.510421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.510402    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad02b257-14a7-415f-835e-2bb332e7e1f1-tmp\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.510491 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.510471    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad02b257-14a7-415f-835e-2bb332e7e1f1-manifests\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.512758 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.512731    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad02b257-14a7-415f-835e-2bb332e7e1f1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.512861 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.512735    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad02b257-14a7-415f-835e-2bb332e7e1f1-tmp\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.512861 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.512815    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad02b257-14a7-415f-835e-2bb332e7e1f1-webhook-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.513820 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.513799    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad02b257-14a7-415f-835e-2bb332e7e1f1-metrics-cert\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.520983 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.520957    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"]
Nov 02 15:01:03.570693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.570648    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 15:01:03.595707 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.595673    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl7s2\" (UniqueName: \"kubernetes.io/projected/ad02b257-14a7-415f-835e-2bb332e7e1f1-kube-api-access-sl7s2\") pod \"opendatahub-operator-controller-manager-55fb46c94b-g6gtf\" (UID: \"ad02b257-14a7-415f-835e-2bb332e7e1f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:03.719572 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:03.719477    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:04.247883 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:04.247857    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"]
Nov 02 15:01:04.248463 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:01:04.248430    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad02b257_14a7_415f_835e_2bb332e7e1f1.slice/crio-0ddba44dd42545e6abbb25453051f8a6726c407a653e0f40422ba434516e25aa WatchSource:0}: Error finding container 0ddba44dd42545e6abbb25453051f8a6726c407a653e0f40422ba434516e25aa: Status 404 returned error can't find the container with id 0ddba44dd42545e6abbb25453051f8a6726c407a653e0f40422ba434516e25aa
Nov 02 15:01:05.019545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:05.019506    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf" event={"ID":"ad02b257-14a7-415f-835e-2bb332e7e1f1","Type":"ContainerStarted","Data":"0ddba44dd42545e6abbb25453051f8a6726c407a653e0f40422ba434516e25aa"}
Nov 02 15:01:06.182245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:06.182211    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb"]
Nov 02 15:01:06.214927 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:06.214894    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb"
Nov 02 15:01:06.229910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:06.229881    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s89nr\" (UniqueName: \"kubernetes.io/projected/113a5664-c697-4538-baf5-4aa46e2b54bc-kube-api-access-s89nr\") pod \"limitador-operator-controller-manager-5778bb4449-dxtzb\" (UID: \"113a5664-c697-4538-baf5-4aa46e2b54bc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb"
Nov 02 15:01:06.246609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:06.246573    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb"]
Nov 02 15:01:06.293622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:06.293588    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-crvt4\""
Nov 02 15:01:06.331018 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:06.330979    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s89nr\" (UniqueName: \"kubernetes.io/projected/113a5664-c697-4538-baf5-4aa46e2b54bc-kube-api-access-s89nr\") pod \"limitador-operator-controller-manager-5778bb4449-dxtzb\" (UID: \"113a5664-c697-4538-baf5-4aa46e2b54bc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb"
Nov 02 15:01:06.396735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:06.396705    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s89nr\" (UniqueName: \"kubernetes.io/projected/113a5664-c697-4538-baf5-4aa46e2b54bc-kube-api-access-s89nr\") pod \"limitador-operator-controller-manager-5778bb4449-dxtzb\" (UID: \"113a5664-c697-4538-baf5-4aa46e2b54bc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb"
Nov 02 15:01:06.571776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:06.571739    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb"
Nov 02 15:01:06.980887 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:06.980809    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb"]
Nov 02 15:01:07.148607 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:01:07.148562    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113a5664_c697_4538_baf5_4aa46e2b54bc.slice/crio-c7e1b876a2ef9ea953945d1f3d504d532c6b9d7e5a33b0c6c2952b2cc8332e2f WatchSource:0}: Error finding container c7e1b876a2ef9ea953945d1f3d504d532c6b9d7e5a33b0c6c2952b2cc8332e2f: Status 404 returned error can't find the container with id c7e1b876a2ef9ea953945d1f3d504d532c6b9d7e5a33b0c6c2952b2cc8332e2f
Nov 02 15:01:08.036327 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:08.036288    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb" event={"ID":"113a5664-c697-4538-baf5-4aa46e2b54bc","Type":"ContainerStarted","Data":"c7e1b876a2ef9ea953945d1f3d504d532c6b9d7e5a33b0c6c2952b2cc8332e2f"}
Nov 02 15:01:08.037994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:08.037968    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd" event={"ID":"8bdb971a-ff5a-4c55-aa72-0de03bb0a4f8","Type":"ContainerStarted","Data":"3d9155e295c2c15941a753e110590e45897d975924dee091ba1b18fd6e0931b3"}
Nov 02 15:01:08.038135 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:08.038108    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd"
Nov 02 15:01:08.169793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:08.169738    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd" podStartSLOduration=2.757728985 podStartE2EDuration="7.169722474s" totalImagesPullingTime="4.411993489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:08.169118727 +0000 UTC m=+275.761735527" watchObservedRunningTime="2026-11-02 15:01:08.169722474 +0000 UTC m=+275.762339272"
Nov 02 15:01:09.044725 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:09.044681    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf" event={"ID":"ad02b257-14a7-415f-835e-2bb332e7e1f1","Type":"ContainerStarted","Data":"37642e218fcfec55cf50132ab9a597523b67f3e2f0103ff81e0ebaadd89dea12"}
Nov 02 15:01:10.056718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:10.056673    4891 generic.go:356] "Generic (PLEG): container finished" podID="ad02b257-14a7-415f-835e-2bb332e7e1f1" containerID="37642e218fcfec55cf50132ab9a597523b67f3e2f0103ff81e0ebaadd89dea12" exitCode=0
Nov 02 15:01:10.057181 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:10.056759    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf" event={"ID":"ad02b257-14a7-415f-835e-2bb332e7e1f1","Type":"ContainerDied","Data":"37642e218fcfec55cf50132ab9a597523b67f3e2f0103ff81e0ebaadd89dea12"}
Nov 02 15:01:10.058931 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:10.058901    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb" event={"ID":"113a5664-c697-4538-baf5-4aa46e2b54bc","Type":"ContainerStarted","Data":"936c3846f9bb9303d2a59c218e3f57e566f10c841da102a0a4c12b33db22eed4"}
Nov 02 15:01:10.059059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:10.059043    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb"
Nov 02 15:01:10.344521 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:10.344425    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb" podStartSLOduration=1.746480435 podStartE2EDuration="4.34441185s" totalImagesPullingTime="2.597931415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:10.34269141 +0000 UTC m=+277.935308210" watchObservedRunningTime="2026-11-02 15:01:10.34441185 +0000 UTC m=+277.937028648"
Nov 02 15:01:11.065753 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:11.065708    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf" event={"ID":"ad02b257-14a7-415f-835e-2bb332e7e1f1","Type":"ContainerStarted","Data":"f4943ea52483c967c0c87a444415cea1a1427b889c64223705de69dbeeb23ea2"}
Nov 02 15:01:11.066287 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:11.066263    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:11.253102 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:11.252997    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf" podStartSLOduration=2.691838255 podStartE2EDuration="8.252983589s" totalImagesPullingTime="5.084235279s" totalInitContainerRuntime="476.910055ms" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:11.252062876 +0000 UTC m=+278.844679676" watchObservedRunningTime="2026-11-02 15:01:11.252983589 +0000 UTC m=+278.845600388"
Nov 02 15:01:14.016465 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:14.016430    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hr5x2"
Nov 02 15:01:14.272753 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:01:14.272716    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:01:15.630675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.630635    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl"]
Nov 02 15:01:15.634398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.634379    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl"
Nov 02 15:01:15.661854 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.661823    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl"]
Nov 02 15:01:15.702178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.702148    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z4ld\" (UniqueName: \"kubernetes.io/projected/292954d7-1a5a-49b0-8d04-dafc05e4ef84-kube-api-access-6z4ld\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl\" (UID: \"292954d7-1a5a-49b0-8d04-dafc05e4ef84\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl"
Nov 02 15:01:15.702358 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.702192    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/292954d7-1a5a-49b0-8d04-dafc05e4ef84-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl\" (UID: \"292954d7-1a5a-49b0-8d04-dafc05e4ef84\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl"
Nov 02 15:01:15.708495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.708467    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 02 15:01:15.708495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.708486    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 02 15:01:15.708644 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.708528    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-c2sfb\""
Nov 02 15:01:15.802640 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.802602    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z4ld\" (UniqueName: \"kubernetes.io/projected/292954d7-1a5a-49b0-8d04-dafc05e4ef84-kube-api-access-6z4ld\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl\" (UID: \"292954d7-1a5a-49b0-8d04-dafc05e4ef84\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl"
Nov 02 15:01:15.802796 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.802682    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/292954d7-1a5a-49b0-8d04-dafc05e4ef84-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl\" (UID: \"292954d7-1a5a-49b0-8d04-dafc05e4ef84\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl"
Nov 02 15:01:15.803158 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.803139    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/292954d7-1a5a-49b0-8d04-dafc05e4ef84-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl\" (UID: \"292954d7-1a5a-49b0-8d04-dafc05e4ef84\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl"
Nov 02 15:01:15.865045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.865010    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z4ld\" (UniqueName: \"kubernetes.io/projected/292954d7-1a5a-49b0-8d04-dafc05e4ef84-kube-api-access-6z4ld\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl\" (UID: \"292954d7-1a5a-49b0-8d04-dafc05e4ef84\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl"
Nov 02 15:01:15.944858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:15.944766    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl"
Nov 02 15:01:16.386686 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:16.386658    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl"]
Nov 02 15:01:16.388544 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:01:16.388510    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292954d7_1a5a_49b0_8d04_dafc05e4ef84.slice/crio-0d4b49875d3a27601125ea1fa64f8d3810816cff0e3e0b5f5484649e7cb4e84a WatchSource:0}: Error finding container 0d4b49875d3a27601125ea1fa64f8d3810816cff0e3e0b5f5484649e7cb4e84a: Status 404 returned error can't find the container with id 0d4b49875d3a27601125ea1fa64f8d3810816cff0e3e0b5f5484649e7cb4e84a
Nov 02 15:01:17.096794 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:17.096755    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl" event={"ID":"292954d7-1a5a-49b0-8d04-dafc05e4ef84","Type":"ContainerStarted","Data":"0d4b49875d3a27601125ea1fa64f8d3810816cff0e3e0b5f5484649e7cb4e84a"}
Nov 02 15:01:19.047731 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:19.047704    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9xdqd"
Nov 02 15:01:21.069013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:21.068982    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dxtzb"
Nov 02 15:01:21.118497 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:21.118456    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl" event={"ID":"292954d7-1a5a-49b0-8d04-dafc05e4ef84","Type":"ContainerStarted","Data":"6340ff0b49318ae6d3313f6b964c5249e9a82e837b2082c8c5374599c3144e14"}
Nov 02 15:01:21.313692 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:21.313641    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-nwbwl" podStartSLOduration=1.979045298 podStartE2EDuration="6.313625646s" totalImagesPullingTime="4.334580348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:21.31116768 +0000 UTC m=+288.903784495" watchObservedRunningTime="2026-11-02 15:01:21.313625646 +0000 UTC m=+288.906242442"
Nov 02 15:01:23.078411 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:23.078378    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55fb46c94b-g6gtf"
Nov 02 15:01:32.526513 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:32.526480    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:01:32.527205 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:32.527187    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:01:32.527535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:32.527523    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:01:32.528199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:32.528180    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:01:32.541915 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:32.541895    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:01:32.542802 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:32.542785    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:01:42.828810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:42.828732    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"]
Nov 02 15:01:42.885364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:42.885325    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"
Nov 02 15:01:42.906827 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:42.906796    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"]
Nov 02 15:01:42.971290 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:42.971264    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 15:01:42.990436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:42.990405    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 15:01:42.990436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:42.990405    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qwfj6\""
Nov 02 15:01:43.032644 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.032609    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb5c56a-7799-400c-a789-bdb39d815648-tls-certs\") pod \"kube-auth-proxy-d9588f984-vxkqn\" (UID: \"6eb5c56a-7799-400c-a789-bdb39d815648\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"
Nov 02 15:01:43.032831 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.032658    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6eb5c56a-7799-400c-a789-bdb39d815648-tmp\") pod \"kube-auth-proxy-d9588f984-vxkqn\" (UID: \"6eb5c56a-7799-400c-a789-bdb39d815648\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"
Nov 02 15:01:43.032831 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.032754    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j22cv\" (UniqueName: \"kubernetes.io/projected/6eb5c56a-7799-400c-a789-bdb39d815648-kube-api-access-j22cv\") pod \"kube-auth-proxy-d9588f984-vxkqn\" (UID: \"6eb5c56a-7799-400c-a789-bdb39d815648\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"
Nov 02 15:01:43.133654 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.133569    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6eb5c56a-7799-400c-a789-bdb39d815648-tmp\") pod \"kube-auth-proxy-d9588f984-vxkqn\" (UID: \"6eb5c56a-7799-400c-a789-bdb39d815648\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"
Nov 02 15:01:43.133654 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.133630    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j22cv\" (UniqueName: \"kubernetes.io/projected/6eb5c56a-7799-400c-a789-bdb39d815648-kube-api-access-j22cv\") pod \"kube-auth-proxy-d9588f984-vxkqn\" (UID: \"6eb5c56a-7799-400c-a789-bdb39d815648\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"
Nov 02 15:01:43.133836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.133695    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb5c56a-7799-400c-a789-bdb39d815648-tls-certs\") pod \"kube-auth-proxy-d9588f984-vxkqn\" (UID: \"6eb5c56a-7799-400c-a789-bdb39d815648\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"
Nov 02 15:01:43.135959 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.135935    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6eb5c56a-7799-400c-a789-bdb39d815648-tmp\") pod \"kube-auth-proxy-d9588f984-vxkqn\" (UID: \"6eb5c56a-7799-400c-a789-bdb39d815648\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"
Nov 02 15:01:43.136335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.136294    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb5c56a-7799-400c-a789-bdb39d815648-tls-certs\") pod \"kube-auth-proxy-d9588f984-vxkqn\" (UID: \"6eb5c56a-7799-400c-a789-bdb39d815648\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"
Nov 02 15:01:43.194584 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.194550    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j22cv\" (UniqueName: \"kubernetes.io/projected/6eb5c56a-7799-400c-a789-bdb39d815648-kube-api-access-j22cv\") pod \"kube-auth-proxy-d9588f984-vxkqn\" (UID: \"6eb5c56a-7799-400c-a789-bdb39d815648\") " pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"
Nov 02 15:01:43.196146 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.196125    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"
Nov 02 15:01:43.331739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.331693    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"]
Nov 02 15:01:43.334241 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.334216    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.386396 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.386306    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-x2959\""
Nov 02 15:01:43.396505 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.396471    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"]
Nov 02 15:01:43.436794 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.436755    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.436964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.436813    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.436964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.436914    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.436964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.436944    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c22dde-651b-4384-948c-d6467611acf5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.437094 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.436983    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c22dde-651b-4384-948c-d6467611acf5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.437094 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.437080    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c22dde-651b-4384-948c-d6467611acf5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.437174 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.437120    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zt9c\" (UniqueName: \"kubernetes.io/projected/44c22dde-651b-4384-948c-d6467611acf5-kube-api-access-4zt9c\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.437174 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.437149    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.437250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.437232    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.537722 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.537683    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c22dde-651b-4384-948c-d6467611acf5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.537927 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.537731    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c22dde-651b-4384-948c-d6467611acf5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.537927 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.537771    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zt9c\" (UniqueName: \"kubernetes.io/projected/44c22dde-651b-4384-948c-d6467611acf5-kube-api-access-4zt9c\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.537927 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.537806    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.537927 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.537835    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.537927 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.537869    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.537927 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.537911    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.538237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.537946    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.538237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.537972    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c22dde-651b-4384-948c-d6467611acf5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.538470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.538424    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.538550 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.538485    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.538611 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.538540    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.538611 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.538575    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.538815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.538792    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c22dde-651b-4384-948c-d6467611acf5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.540144 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.540123    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c22dde-651b-4384-948c-d6467611acf5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.540287 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.540268    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c22dde-651b-4384-948c-d6467611acf5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.571402 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.571380    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d9588f984-vxkqn"]
Nov 02 15:01:43.573239 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:01:43.573210    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb5c56a_7799_400c_a789_bdb39d815648.slice/crio-04b2568aab9518f69e5ce284ac45c090a7b8d073fd59d714853421ee14212587 WatchSource:0}: Error finding container 04b2568aab9518f69e5ce284ac45c090a7b8d073fd59d714853421ee14212587: Status 404 returned error can't find the container with id 04b2568aab9518f69e5ce284ac45c090a7b8d073fd59d714853421ee14212587
Nov 02 15:01:43.575040 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.575025    4891 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:01:43.598989 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.598964    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c22dde-651b-4384-948c-d6467611acf5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.599110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.599092    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zt9c\" (UniqueName: \"kubernetes.io/projected/44c22dde-651b-4384-948c-d6467611acf5-kube-api-access-4zt9c\") pod \"data-science-gateway-data-science-gateway-class-6b748b65857h8ws\" (UID: \"44c22dde-651b-4384-948c-d6467611acf5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:43.647792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:43.647724    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:44.082231 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:44.082205    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"]
Nov 02 15:01:44.084683 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:01:44.084652    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c22dde_651b_4384_948c_d6467611acf5.slice/crio-a9826e057f19fcb264848ad4dc2119d3321db436472f89cac7fffc44f6285d36 WatchSource:0}: Error finding container a9826e057f19fcb264848ad4dc2119d3321db436472f89cac7fffc44f6285d36: Status 404 returned error can't find the container with id a9826e057f19fcb264848ad4dc2119d3321db436472f89cac7fffc44f6285d36
Nov 02 15:01:44.086782 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:44.086744    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:01:44.086888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:44.086830    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:01:44.086888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:44.086876    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:01:44.242185 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:44.242144    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws" event={"ID":"44c22dde-651b-4384-948c-d6467611acf5","Type":"ContainerStarted","Data":"9e442a29b9b834719aa04c4d8219841bf15ab25284b8f6f7e6bf6a770c0501d6"}
Nov 02 15:01:44.242401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:44.242211    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws" event={"ID":"44c22dde-651b-4384-948c-d6467611acf5","Type":"ContainerStarted","Data":"a9826e057f19fcb264848ad4dc2119d3321db436472f89cac7fffc44f6285d36"}
Nov 02 15:01:44.243714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:44.243689    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn" event={"ID":"6eb5c56a-7799-400c-a789-bdb39d815648","Type":"ContainerStarted","Data":"04b2568aab9518f69e5ce284ac45c090a7b8d073fd59d714853421ee14212587"}
Nov 02 15:01:44.274253 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:01:44.274213    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:01:44.366290 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:44.366201    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws" podStartSLOduration=1.366186444 podStartE2EDuration="1.366186444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:44.364059825 +0000 UTC m=+311.956676626" watchObservedRunningTime="2026-11-02 15:01:44.366186444 +0000 UTC m=+311.958803243"
Nov 02 15:01:44.647986 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:44.647900    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:45.653443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:45.653408    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:46.254382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:46.254352    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:46.255565 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:46.255538    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65857h8ws"
Nov 02 15:01:47.260780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:47.260736    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn" event={"ID":"6eb5c56a-7799-400c-a789-bdb39d815648","Type":"ContainerStarted","Data":"a60e16fa33a6dbac694b36810001950c360a45bcb55979832b522b988b1d68eb"}
Nov 02 15:01:47.384207 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:01:47.384154    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d9588f984-vxkqn" podStartSLOduration=2.679320856 podStartE2EDuration="5.384140036s" totalImagesPullingTime="2.70481918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:47.381577622 +0000 UTC m=+314.974194422" watchObservedRunningTime="2026-11-02 15:01:47.384140036 +0000 UTC m=+314.976756834"
Nov 02 15:02:14.272922 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:02:14.272879    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:02:44.272198 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:02:44.272158    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:03:14.273791 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:03:14.273707    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:03:15.762368 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.762333    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dtrlb"]
Nov 02 15:03:15.765732 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.765711    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb"
Nov 02 15:03:15.867348 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.867259    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 02 15:03:15.867576 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.867347    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xt64w\""
Nov 02 15:03:15.867576 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.867502    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 15:03:15.867576 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.867541    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 15:03:15.869697 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.869671    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dtrlb"]
Nov 02 15:03:15.875984 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.875959    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"]
Nov 02 15:03:15.879183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.879163    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"
Nov 02 15:03:15.898443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.898415    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c97de6e4-caef-4acb-b67e-04160d6d60ca-cert\") pod \"kserve-controller-manager-776cc66d47-dtrlb\" (UID: \"c97de6e4-caef-4acb-b67e-04160d6d60ca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb"
Nov 02 15:03:15.898579 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.898477    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-947kk\" (UniqueName: \"kubernetes.io/projected/c97de6e4-caef-4acb-b67e-04160d6d60ca-kube-api-access-947kk\") pod \"kserve-controller-manager-776cc66d47-dtrlb\" (UID: \"c97de6e4-caef-4acb-b67e-04160d6d60ca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb"
Nov 02 15:03:15.931539 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.931502    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 02 15:03:15.931718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.931551    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-zwb6m\""
Nov 02 15:03:15.937993 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.937964    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"]
Nov 02 15:03:15.999961 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.999929    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c97de6e4-caef-4acb-b67e-04160d6d60ca-cert\") pod \"kserve-controller-manager-776cc66d47-dtrlb\" (UID: \"c97de6e4-caef-4acb-b67e-04160d6d60ca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb"
Nov 02 15:03:16.000125 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:15.999970    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9965d\" (UID: \"ba4a525d-d390-41d2-9f85-5ca14dcba2c1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"
Nov 02 15:03:16.000125 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.000009    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h45lw\" (UniqueName: \"kubernetes.io/projected/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-kube-api-access-h45lw\") pod \"llmisvc-controller-manager-869d5dcc5d-9965d\" (UID: \"ba4a525d-d390-41d2-9f85-5ca14dcba2c1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"
Nov 02 15:03:16.000198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.000136    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-947kk\" (UniqueName: \"kubernetes.io/projected/c97de6e4-caef-4acb-b67e-04160d6d60ca-kube-api-access-947kk\") pod \"kserve-controller-manager-776cc66d47-dtrlb\" (UID: \"c97de6e4-caef-4acb-b67e-04160d6d60ca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb"
Nov 02 15:03:16.002517 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.002494    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c97de6e4-caef-4acb-b67e-04160d6d60ca-cert\") pod \"kserve-controller-manager-776cc66d47-dtrlb\" (UID: \"c97de6e4-caef-4acb-b67e-04160d6d60ca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb"
Nov 02 15:03:16.065653 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.065623    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-947kk\" (UniqueName: \"kubernetes.io/projected/c97de6e4-caef-4acb-b67e-04160d6d60ca-kube-api-access-947kk\") pod \"kserve-controller-manager-776cc66d47-dtrlb\" (UID: \"c97de6e4-caef-4acb-b67e-04160d6d60ca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb"
Nov 02 15:03:16.078398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.078377    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb"
Nov 02 15:03:16.101741 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.101707    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9965d\" (UID: \"ba4a525d-d390-41d2-9f85-5ca14dcba2c1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"
Nov 02 15:03:16.101868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.101784    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h45lw\" (UniqueName: \"kubernetes.io/projected/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-kube-api-access-h45lw\") pod \"llmisvc-controller-manager-869d5dcc5d-9965d\" (UID: \"ba4a525d-d390-41d2-9f85-5ca14dcba2c1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"
Nov 02 15:03:16.104802 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.104772    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9965d\" (UID: \"ba4a525d-d390-41d2-9f85-5ca14dcba2c1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"
Nov 02 15:03:16.166829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.166801    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h45lw\" (UniqueName: \"kubernetes.io/projected/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-kube-api-access-h45lw\") pod \"llmisvc-controller-manager-869d5dcc5d-9965d\" (UID: \"ba4a525d-d390-41d2-9f85-5ca14dcba2c1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"
Nov 02 15:03:16.192610 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.192578    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"
Nov 02 15:03:16.460352 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.457345    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dtrlb"]
Nov 02 15:03:16.567765 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.567659    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"]
Nov 02 15:03:16.569786 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:03:16.569756    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podba4a525d_d390_41d2_9f85_5ca14dcba2c1.slice/crio-09fe03da4a9ff813473402b1c3e087c012c1b5df7c27d7362a3740841cb14c16 WatchSource:0}: Error finding container 09fe03da4a9ff813473402b1c3e087c012c1b5df7c27d7362a3740841cb14c16: Status 404 returned error can't find the container with id 09fe03da4a9ff813473402b1c3e087c012c1b5df7c27d7362a3740841cb14c16
Nov 02 15:03:16.713979 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.713881    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb" event={"ID":"c97de6e4-caef-4acb-b67e-04160d6d60ca","Type":"ContainerStarted","Data":"b1af0432f620e1193b4c7691b40b08845990d088ed7c897801ec364d36bc3398"}
Nov 02 15:03:16.715065 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:16.715043    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d" event={"ID":"ba4a525d-d390-41d2-9f85-5ca14dcba2c1","Type":"ContainerStarted","Data":"09fe03da4a9ff813473402b1c3e087c012c1b5df7c27d7362a3740841cb14c16"}
Nov 02 15:03:19.736214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:19.736176    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d" event={"ID":"ba4a525d-d390-41d2-9f85-5ca14dcba2c1","Type":"ContainerStarted","Data":"f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d"}
Nov 02 15:03:19.736657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:19.736264    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"
Nov 02 15:03:19.737852 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:19.737819    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb" event={"ID":"c97de6e4-caef-4acb-b67e-04160d6d60ca","Type":"ContainerStarted","Data":"4570833f8a3c8b299e43c73f5524555b18d22f9bde5320b4c578630bf138623a"}
Nov 02 15:03:19.737976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:19.737957    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb"
Nov 02 15:03:19.982561 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:19.982502    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb" podStartSLOduration=2.32261131 podStartE2EDuration="4.982486698s" totalImagesPullingTime="2.659875388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:19.98100226 +0000 UTC m=+407.573619059" watchObservedRunningTime="2026-11-02 15:03:19.982486698 +0000 UTC m=+407.575103540"
Nov 02 15:03:19.983303 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:19.983277    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d" podStartSLOduration=2.434511325 podStartE2EDuration="4.983270878s" totalImagesPullingTime="2.548759553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:19.865590601 +0000 UTC m=+407.458207401" watchObservedRunningTime="2026-11-02 15:03:19.983270878 +0000 UTC m=+407.575887713"
Nov 02 15:03:24.653566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:24.653530    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58f7f45598-hlfxn"]
Nov 02 15:03:24.657231 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:24.657182    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58f7f45598-hlfxn"
Nov 02 15:03:24.711364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:24.711322    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0f9352b6-4fd7-4e89-af11-160ee84f8861-tls-cert\") pod \"authorino-58f7f45598-hlfxn\" (UID: \"0f9352b6-4fd7-4e89-af11-160ee84f8861\") " pod="kuadrant-system/authorino-58f7f45598-hlfxn"
Nov 02 15:03:24.712127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:24.712054    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww6dp\" (UniqueName: \"kubernetes.io/projected/0f9352b6-4fd7-4e89-af11-160ee84f8861-kube-api-access-ww6dp\") pod \"authorino-58f7f45598-hlfxn\" (UID: \"0f9352b6-4fd7-4e89-af11-160ee84f8861\") " pod="kuadrant-system/authorino-58f7f45598-hlfxn"
Nov 02 15:03:24.714981 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:24.714954    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58f7f45598-hlfxn"]
Nov 02 15:03:24.761934 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:24.761900    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 15:03:24.761934 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:24.761903    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rnjnp\""
Nov 02 15:03:24.812994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:24.812966    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww6dp\" (UniqueName: \"kubernetes.io/projected/0f9352b6-4fd7-4e89-af11-160ee84f8861-kube-api-access-ww6dp\") pod \"authorino-58f7f45598-hlfxn\" (UID: \"0f9352b6-4fd7-4e89-af11-160ee84f8861\") " pod="kuadrant-system/authorino-58f7f45598-hlfxn"
Nov 02 15:03:24.813172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:24.813030    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0f9352b6-4fd7-4e89-af11-160ee84f8861-tls-cert\") pod \"authorino-58f7f45598-hlfxn\" (UID: \"0f9352b6-4fd7-4e89-af11-160ee84f8861\") " pod="kuadrant-system/authorino-58f7f45598-hlfxn"
Nov 02 15:03:24.815429 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:24.815410    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0f9352b6-4fd7-4e89-af11-160ee84f8861-tls-cert\") pod \"authorino-58f7f45598-hlfxn\" (UID: \"0f9352b6-4fd7-4e89-af11-160ee84f8861\") " pod="kuadrant-system/authorino-58f7f45598-hlfxn"
Nov 02 15:03:24.873849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:24.873820    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww6dp\" (UniqueName: \"kubernetes.io/projected/0f9352b6-4fd7-4e89-af11-160ee84f8861-kube-api-access-ww6dp\") pod \"authorino-58f7f45598-hlfxn\" (UID: \"0f9352b6-4fd7-4e89-af11-160ee84f8861\") " pod="kuadrant-system/authorino-58f7f45598-hlfxn"
Nov 02 15:03:24.973009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:24.972926    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58f7f45598-hlfxn"
Nov 02 15:03:25.350514 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:25.350463    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58f7f45598-hlfxn"]
Nov 02 15:03:25.353350 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:03:25.353320    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f9352b6_4fd7_4e89_af11_160ee84f8861.slice/crio-34f5a31ca05ad030b4151853b6695b760c7fb9a417e789cd259da7e317b58219 WatchSource:0}: Error finding container 34f5a31ca05ad030b4151853b6695b760c7fb9a417e789cd259da7e317b58219: Status 404 returned error can't find the container with id 34f5a31ca05ad030b4151853b6695b760c7fb9a417e789cd259da7e317b58219
Nov 02 15:03:25.793912 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:25.793867    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58f7f45598-hlfxn" event={"ID":"0f9352b6-4fd7-4e89-af11-160ee84f8861","Type":"ContainerStarted","Data":"34f5a31ca05ad030b4151853b6695b760c7fb9a417e789cd259da7e317b58219"}
Nov 02 15:03:27.813129 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:27.813044    4891 generic.go:356] "Generic (PLEG): container finished" podID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerID="540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de" exitCode=0
Nov 02 15:03:27.813129 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:27.813108    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" event={"ID":"adcd5628-e804-4d09-aa88-9af4a730ef2c","Type":"ContainerDied","Data":"540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de"}
Nov 02 15:03:27.813681 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:27.813530    4891 scope.go:122] "RemoveContainer" containerID="540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de"
Nov 02 15:03:27.814807 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:27.814767    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:03:27.814886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:27.814849    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:03:27.814933 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:27.814883    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:03:28.669452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:28.669411    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:03:28.824163 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:28.824126    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" event={"ID":"adcd5628-e804-4d09-aa88-9af4a730ef2c","Type":"ContainerStarted","Data":"9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8"}
Nov 02 15:03:29.155334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:29.151089    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:03:29.162674 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:29.162294    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 02 15:03:29.162674 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:29.162386    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 02 15:03:29.838339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:29.837814    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58f7f45598-hlfxn" event={"ID":"0f9352b6-4fd7-4e89-af11-160ee84f8861","Type":"ContainerStarted","Data":"9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a"}
Nov 02 15:03:29.956336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:29.956251    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-58f7f45598-hlfxn" podStartSLOduration=2.405663679 podStartE2EDuration="5.956229131s" totalImagesPullingTime="3.550565452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:29.952684166 +0000 UTC m=+417.545300966" watchObservedRunningTime="2026-11-02 15:03:29.956229131 +0000 UTC m=+417.548845936"
Nov 02 15:03:30.170629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:30.170532    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 02 15:03:30.170629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:30.170610    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 02 15:03:31.149790 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:31.149746    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 02 15:03:31.161468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:31.161418    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 02 15:03:32.160347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:32.159763    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 02 15:03:32.160347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:32.159830    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 02 15:03:33.163337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:33.157766    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 02 15:03:33.163337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:33.157833    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 02 15:03:34.153301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:34.153269    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:03:34.153567 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:34.153547    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:03:34.154199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:34.154179    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:03:44.273649 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:03:44.273609    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:03:50.746194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:50.746160    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"
Nov 02 15:03:50.749219 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:03:50.749200    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-dtrlb"
Nov 02 15:04:14.273527 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:04:14.273483    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:04:31.550344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:31.550287    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"]
Nov 02 15:04:31.554002 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:31.553983    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"
Nov 02 15:04:31.602438 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:31.602406    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgx9d\" (UniqueName: \"kubernetes.io/projected/892e6d1a-7a1a-4f5f-bfac-456c20481bca-kube-api-access-vgx9d\") pod \"llmisvc-controller-manager-6466c554d8-fldg9\" (UID: \"892e6d1a-7a1a-4f5f-bfac-456c20481bca\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"
Nov 02 15:04:31.602578 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:31.602444    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/892e6d1a-7a1a-4f5f-bfac-456c20481bca-cert\") pod \"llmisvc-controller-manager-6466c554d8-fldg9\" (UID: \"892e6d1a-7a1a-4f5f-bfac-456c20481bca\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"
Nov 02 15:04:31.611555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:31.611530    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"]
Nov 02 15:04:31.703214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:31.703176    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/892e6d1a-7a1a-4f5f-bfac-456c20481bca-cert\") pod \"llmisvc-controller-manager-6466c554d8-fldg9\" (UID: \"892e6d1a-7a1a-4f5f-bfac-456c20481bca\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"
Nov 02 15:04:31.703440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:31.703334    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgx9d\" (UniqueName: \"kubernetes.io/projected/892e6d1a-7a1a-4f5f-bfac-456c20481bca-kube-api-access-vgx9d\") pod \"llmisvc-controller-manager-6466c554d8-fldg9\" (UID: \"892e6d1a-7a1a-4f5f-bfac-456c20481bca\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"
Nov 02 15:04:31.705603 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:31.705583    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/892e6d1a-7a1a-4f5f-bfac-456c20481bca-cert\") pod \"llmisvc-controller-manager-6466c554d8-fldg9\" (UID: \"892e6d1a-7a1a-4f5f-bfac-456c20481bca\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"
Nov 02 15:04:31.764333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:31.764272    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgx9d\" (UniqueName: \"kubernetes.io/projected/892e6d1a-7a1a-4f5f-bfac-456c20481bca-kube-api-access-vgx9d\") pod \"llmisvc-controller-manager-6466c554d8-fldg9\" (UID: \"892e6d1a-7a1a-4f5f-bfac-456c20481bca\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"
Nov 02 15:04:31.867522 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:31.867428    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"
Nov 02 15:04:32.240244 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:32.240220    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"]
Nov 02 15:04:32.242610 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:04:32.242576    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892e6d1a_7a1a_4f5f_bfac_456c20481bca.slice/crio-9ddf2936d2d640762aaace65d31409003e4bf444a6bd06dd3721f04298da3fb1 WatchSource:0}: Error finding container 9ddf2936d2d640762aaace65d31409003e4bf444a6bd06dd3721f04298da3fb1: Status 404 returned error can't find the container with id 9ddf2936d2d640762aaace65d31409003e4bf444a6bd06dd3721f04298da3fb1
Nov 02 15:04:33.183505 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:33.183464    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9" event={"ID":"892e6d1a-7a1a-4f5f-bfac-456c20481bca","Type":"ContainerStarted","Data":"4210bcede08d37838484feb139c7d8ad72cd360a9d6b202173ff2b9ef4affaa7"}
Nov 02 15:04:33.183905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:33.183698    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"
Nov 02 15:04:33.183905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:33.183738    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9" event={"ID":"892e6d1a-7a1a-4f5f-bfac-456c20481bca","Type":"ContainerStarted","Data":"9ddf2936d2d640762aaace65d31409003e4bf444a6bd06dd3721f04298da3fb1"}
Nov 02 15:04:33.300621 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:04:33.300569    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9" podStartSLOduration=1.537345773 podStartE2EDuration="2.300555126s" totalImagesPullingTime="763.209353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:33.299283255 +0000 UTC m=+480.891900055" watchObservedRunningTime="2026-11-02 15:04:33.300555126 +0000 UTC m=+480.893171924"
Nov 02 15:04:44.273787 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:04:44.273703    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:05:04.191729 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:04.191696    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fldg9"
Nov 02 15:05:04.361328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:04.361280    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"]
Nov 02 15:05:04.361544 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:04.361517    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d" podUID="ba4a525d-d390-41d2-9f85-5ca14dcba2c1" containerName="manager" containerID="cri-o://f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d" gracePeriod=30
Nov 02 15:05:04.613301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:04.613279    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"
Nov 02 15:05:04.792755 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:04.792725    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-kube-api-access-h45lw\" (UniqueName: \"kubernetes.io/projected/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-kube-api-access-h45lw\") pod \"ba4a525d-d390-41d2-9f85-5ca14dcba2c1\" (UID: \"ba4a525d-d390-41d2-9f85-5ca14dcba2c1\") "
Nov 02 15:05:04.792938 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:04.792898    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-cert\" (UniqueName: \"kubernetes.io/secret/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-cert\") pod \"ba4a525d-d390-41d2-9f85-5ca14dcba2c1\" (UID: \"ba4a525d-d390-41d2-9f85-5ca14dcba2c1\") "
Nov 02 15:05:04.794939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:04.794913    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-kube-api-access-h45lw" pod "ba4a525d-d390-41d2-9f85-5ca14dcba2c1" (UID: "ba4a525d-d390-41d2-9f85-5ca14dcba2c1"). InnerVolumeSpecName "kube-api-access-h45lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:05:04.794939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:04.794926    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-cert" pod "ba4a525d-d390-41d2-9f85-5ca14dcba2c1" (UID: "ba4a525d-d390-41d2-9f85-5ca14dcba2c1"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:05:04.894265 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:04.894229    4891 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:05:04.894265 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:04.894262    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h45lw\" (UniqueName: \"kubernetes.io/projected/ba4a525d-d390-41d2-9f85-5ca14dcba2c1-kube-api-access-h45lw\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:05:05.359362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:05.359301    4891 generic.go:356] "Generic (PLEG): container finished" podID="ba4a525d-d390-41d2-9f85-5ca14dcba2c1" containerID="f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d" exitCode=0
Nov 02 15:05:05.359830 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:05.359362    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d" event={"ID":"ba4a525d-d390-41d2-9f85-5ca14dcba2c1","Type":"ContainerDied","Data":"f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d"}
Nov 02 15:05:05.359830 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:05.359418    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d" event={"ID":"ba4a525d-d390-41d2-9f85-5ca14dcba2c1","Type":"ContainerDied","Data":"09fe03da4a9ff813473402b1c3e087c012c1b5df7c27d7362a3740841cb14c16"}
Nov 02 15:05:05.359830 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:05.359440    4891 scope.go:122] "RemoveContainer" containerID="f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d"
Nov 02 15:05:05.359830 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:05.359370    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"
Nov 02 15:05:05.369211 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:05.369030    4891 scope.go:122] "RemoveContainer" containerID="f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d"
Nov 02 15:05:05.369346 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:05:05.369306    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d\": container with ID starting with f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d not found: ID does not exist" containerID="f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d"
Nov 02 15:05:05.369397 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:05.369357    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d"} err="failed to get container status \"f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d\": rpc error: code = NotFound desc = could not find container \"f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d\": container with ID starting with f03d62c5ab698745f3ef2f24860ce83bfb88d01a11f6efb7b80588ad1e7faa9d not found: ID does not exist"
Nov 02 15:05:05.526469 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:05.526435    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"]
Nov 02 15:05:05.576503 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:05.576471    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9965d"]
Nov 02 15:05:06.520645 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:05:06.520607    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4a525d-d390-41d2-9f85-5ca14dcba2c1" path="/var/lib/kubelet/pods/ba4a525d-d390-41d2-9f85-5ca14dcba2c1/volumes"
Nov 02 15:05:14.273191 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:05:14.273151    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:05:44.282814 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:05:44.282767    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:06:13.724576 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:13.724494    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c96f6ccb5-dfw9m"]
Nov 02 15:06:13.725037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:13.724873    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ba4a525d-d390-41d2-9f85-5ca14dcba2c1" containerName="manager"
Nov 02 15:06:13.725037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:13.724888    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4a525d-d390-41d2-9f85-5ca14dcba2c1" containerName="manager"
Nov 02 15:06:13.725037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:13.724991    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4a525d-d390-41d2-9f85-5ca14dcba2c1" containerName="manager"
Nov 02 15:06:13.728183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:13.728160    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c96f6ccb5-dfw9m"
Nov 02 15:06:13.785885 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:13.785859    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c96f6ccb5-dfw9m"]
Nov 02 15:06:13.812510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:13.812479    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dea5d7e8-792d-4444-9707-6e539828c206-tls-cert\") pod \"authorino-7c96f6ccb5-dfw9m\" (UID: \"dea5d7e8-792d-4444-9707-6e539828c206\") " pod="kuadrant-system/authorino-7c96f6ccb5-dfw9m"
Nov 02 15:06:13.812680 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:13.812516    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhpt\" (UniqueName: \"kubernetes.io/projected/dea5d7e8-792d-4444-9707-6e539828c206-kube-api-access-klhpt\") pod \"authorino-7c96f6ccb5-dfw9m\" (UID: \"dea5d7e8-792d-4444-9707-6e539828c206\") " pod="kuadrant-system/authorino-7c96f6ccb5-dfw9m"
Nov 02 15:06:13.913095 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:13.913054    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dea5d7e8-792d-4444-9707-6e539828c206-tls-cert\") pod \"authorino-7c96f6ccb5-dfw9m\" (UID: \"dea5d7e8-792d-4444-9707-6e539828c206\") " pod="kuadrant-system/authorino-7c96f6ccb5-dfw9m"
Nov 02 15:06:13.913095 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:13.913097    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhpt\" (UniqueName: \"kubernetes.io/projected/dea5d7e8-792d-4444-9707-6e539828c206-kube-api-access-klhpt\") pod \"authorino-7c96f6ccb5-dfw9m\" (UID: \"dea5d7e8-792d-4444-9707-6e539828c206\") " pod="kuadrant-system/authorino-7c96f6ccb5-dfw9m"
Nov 02 15:06:13.915513 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:13.915488    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dea5d7e8-792d-4444-9707-6e539828c206-tls-cert\") pod \"authorino-7c96f6ccb5-dfw9m\" (UID: \"dea5d7e8-792d-4444-9707-6e539828c206\") " pod="kuadrant-system/authorino-7c96f6ccb5-dfw9m"
Nov 02 15:06:13.978535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:13.978467    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhpt\" (UniqueName: \"kubernetes.io/projected/dea5d7e8-792d-4444-9707-6e539828c206-kube-api-access-klhpt\") pod \"authorino-7c96f6ccb5-dfw9m\" (UID: \"dea5d7e8-792d-4444-9707-6e539828c206\") " pod="kuadrant-system/authorino-7c96f6ccb5-dfw9m"
Nov 02 15:06:14.039741 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:14.039703    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c96f6ccb5-dfw9m"
Nov 02 15:06:14.273864 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:06:14.273829    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:06:14.413711 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:14.413472    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c96f6ccb5-dfw9m"]
Nov 02 15:06:14.416044 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:06:14.416001    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddea5d7e8_792d_4444_9707_6e539828c206.slice/crio-047335f885dcfda2658070750a3eda78fb3ffe9ede1cc1f809a2e3cea7db9cc9 WatchSource:0}: Error finding container 047335f885dcfda2658070750a3eda78fb3ffe9ede1cc1f809a2e3cea7db9cc9: Status 404 returned error can't find the container with id 047335f885dcfda2658070750a3eda78fb3ffe9ede1cc1f809a2e3cea7db9cc9
Nov 02 15:06:14.751733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:14.751645    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c96f6ccb5-dfw9m" event={"ID":"dea5d7e8-792d-4444-9707-6e539828c206","Type":"ContainerStarted","Data":"047335f885dcfda2658070750a3eda78fb3ffe9ede1cc1f809a2e3cea7db9cc9"}
Nov 02 15:06:16.764578 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:16.764533    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c96f6ccb5-dfw9m" event={"ID":"dea5d7e8-792d-4444-9707-6e539828c206","Type":"ContainerStarted","Data":"c54b72999762603a916dc319c716ddc84168c0d42e11b6732b293226bd9f00cf"}
Nov 02 15:06:16.879360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:16.879273    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7c96f6ccb5-dfw9m" podStartSLOduration=2.272651672 podStartE2EDuration="3.879253041s" totalImagesPullingTime="1.606601369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:16.879164041 +0000 UTC m=+584.471780834" watchObservedRunningTime="2026-11-02 15:06:16.879253041 +0000 UTC m=+584.471869841"
Nov 02 15:06:16.913901 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:16.913858    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58f7f45598-hlfxn"]
Nov 02 15:06:16.914128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:16.914090    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-58f7f45598-hlfxn" podUID="0f9352b6-4fd7-4e89-af11-160ee84f8861" containerName="authorino" containerID="cri-o://9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a" gracePeriod=30
Nov 02 15:06:17.177278 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.177248    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58f7f45598-hlfxn"
Nov 02 15:06:17.237734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.237694    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9352b6-4fd7-4e89-af11-160ee84f8861-tls-cert\" (UniqueName: \"kubernetes.io/secret/0f9352b6-4fd7-4e89-af11-160ee84f8861-tls-cert\") pod \"0f9352b6-4fd7-4e89-af11-160ee84f8861\" (UID: \"0f9352b6-4fd7-4e89-af11-160ee84f8861\") "
Nov 02 15:06:17.237734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.237740    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9352b6-4fd7-4e89-af11-160ee84f8861-kube-api-access-ww6dp\" (UniqueName: \"kubernetes.io/projected/0f9352b6-4fd7-4e89-af11-160ee84f8861-kube-api-access-ww6dp\") pod \"0f9352b6-4fd7-4e89-af11-160ee84f8861\" (UID: \"0f9352b6-4fd7-4e89-af11-160ee84f8861\") "
Nov 02 15:06:17.239932 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.239901    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9352b6-4fd7-4e89-af11-160ee84f8861-kube-api-access-ww6dp" pod "0f9352b6-4fd7-4e89-af11-160ee84f8861" (UID: "0f9352b6-4fd7-4e89-af11-160ee84f8861"). InnerVolumeSpecName "kube-api-access-ww6dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:06:17.248941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.248904    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9352b6-4fd7-4e89-af11-160ee84f8861-tls-cert" pod "0f9352b6-4fd7-4e89-af11-160ee84f8861" (UID: "0f9352b6-4fd7-4e89-af11-160ee84f8861"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:06:17.342988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.342856    4891 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0f9352b6-4fd7-4e89-af11-160ee84f8861-tls-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:06:17.342988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.342902    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww6dp\" (UniqueName: \"kubernetes.io/projected/0f9352b6-4fd7-4e89-af11-160ee84f8861-kube-api-access-ww6dp\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:06:17.771329 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.771284    4891 generic.go:356] "Generic (PLEG): container finished" podID="0f9352b6-4fd7-4e89-af11-160ee84f8861" containerID="9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a" exitCode=0
Nov 02 15:06:17.771748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.771349    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58f7f45598-hlfxn"
Nov 02 15:06:17.771748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.771348    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58f7f45598-hlfxn" event={"ID":"0f9352b6-4fd7-4e89-af11-160ee84f8861","Type":"ContainerDied","Data":"9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a"}
Nov 02 15:06:17.771748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.771460    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58f7f45598-hlfxn" event={"ID":"0f9352b6-4fd7-4e89-af11-160ee84f8861","Type":"ContainerDied","Data":"34f5a31ca05ad030b4151853b6695b760c7fb9a417e789cd259da7e317b58219"}
Nov 02 15:06:17.771748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.771481    4891 scope.go:122] "RemoveContainer" containerID="9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a"
Nov 02 15:06:17.781414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.781391    4891 scope.go:122] "RemoveContainer" containerID="9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a"
Nov 02 15:06:17.781730 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:06:17.781705    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a\": container with ID starting with 9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a not found: ID does not exist" containerID="9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a"
Nov 02 15:06:17.781791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.781746    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a"} err="failed to get container status \"9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a\": rpc error: code = NotFound desc = could not find container \"9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a\": container with ID starting with 9bf7b4d69ca2bd66b09b557375636a7d5722e07f59557ffa65cf592e0c44f24a not found: ID does not exist"
Nov 02 15:06:17.936581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.936546    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58f7f45598-hlfxn"]
Nov 02 15:06:17.985671 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:17.985635    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58f7f45598-hlfxn"]
Nov 02 15:06:18.521487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.521447    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9352b6-4fd7-4e89-af11-160ee84f8861" path="/var/lib/kubelet/pods/0f9352b6-4fd7-4e89-af11-160ee84f8861/volumes"
Nov 02 15:06:18.827245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.827168    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"]
Nov 02 15:06:18.827630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.827585    4891 cpu_manager.go:414] "containerMap: removing container" podUID="0f9352b6-4fd7-4e89-af11-160ee84f8861" containerName="authorino"
Nov 02 15:06:18.827630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.827599    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9352b6-4fd7-4e89-af11-160ee84f8861" containerName="authorino"
Nov 02 15:06:18.827706 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.827691    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9352b6-4fd7-4e89-af11-160ee84f8861" containerName="authorino"
Nov 02 15:06:18.832118 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.832102    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.856412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.856367    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8mjd\" (UniqueName: \"kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-kube-api-access-v8mjd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.856595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.856494    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.856595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.856535    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.856723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.856642    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.856723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.856682    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.856723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.856716    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.856870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.856757    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.856870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.856791    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.856946 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.856886    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3c572-758e-4a42-92e1-8f10bee50a99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.958119 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.958066    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.958333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.958117    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.958333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.958169    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.958333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.958205    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.958333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.958234    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.958333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.958271    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3c572-758e-4a42-92e1-8f10bee50a99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.958617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.958086    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"]
Nov 02 15:06:18.959433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.958709    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.959433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.958723    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mjd\" (UniqueName: \"kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-kube-api-access-v8mjd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.959433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.958779    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.959433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.958797    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.959433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.958832    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.959433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.959122    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3c572-758e-4a42-92e1-8f10bee50a99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.959433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.959250    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.959433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.959383    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.961904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.961883    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:18.962255 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:18.962231    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:19.019918 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:19.019888    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:19.020075 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:19.020053    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mjd\" (UniqueName: \"kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-kube-api-access-v8mjd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-x44q9\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:19.146283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:19.146202    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:19.584305 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:19.584275    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"]
Nov 02 15:06:19.585691 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:06:19.585660    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce3c572_758e_4a42_92e1_8f10bee50a99.slice/crio-84acb7b8ba54a9c35f5c4b1b36b0af1db830abdfb019bcbf1bf617ee69a73acf WatchSource:0}: Error finding container 84acb7b8ba54a9c35f5c4b1b36b0af1db830abdfb019bcbf1bf617ee69a73acf: Status 404 returned error can't find the container with id 84acb7b8ba54a9c35f5c4b1b36b0af1db830abdfb019bcbf1bf617ee69a73acf
Nov 02 15:06:19.588126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:19.588085    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:06:19.588192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:19.588170    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:06:19.588231 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:19.588210    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:06:19.793584 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:19.793545    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" event={"ID":"1ce3c572-758e-4a42-92e1-8f10bee50a99","Type":"ContainerStarted","Data":"f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7"}
Nov 02 15:06:19.793825 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:19.793625    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" event={"ID":"1ce3c572-758e-4a42-92e1-8f10bee50a99","Type":"ContainerStarted","Data":"84acb7b8ba54a9c35f5c4b1b36b0af1db830abdfb019bcbf1bf617ee69a73acf"}
Nov 02 15:06:19.917707 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:19.917593    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" podStartSLOduration=1.9175738839999998 podStartE2EDuration="1.917573884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:19.915905153 +0000 UTC m=+587.508521953" watchObservedRunningTime="2026-11-02 15:06:19.917573884 +0000 UTC m=+587.510190685"
Nov 02 15:06:20.148792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:20.148757    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:20.149186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:20.149161    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-x44q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:06:20.149372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:20.149347    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 15:06:21.163337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:21.155762    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-x44q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:06:21.163337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:21.155828    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 15:06:22.154336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:22.149797    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-x44q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:06:22.154336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:22.149872    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 15:06:23.156759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:23.156718    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-x44q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:06:23.157385 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:23.157351    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 15:06:24.154333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:24.148760    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-x44q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:06:24.154333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:24.148822    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 15:06:25.166333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:25.160435    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-x44q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:06:25.166333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:25.160497    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 15:06:26.146560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:26.146520    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-x44q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:06:26.146747 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:26.146585    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 15:06:27.903109 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:27.903079    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:27.903607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:27.903480    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:27.904225 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:27.904207    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:06:32.566284 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:32.566248    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:06:32.567129 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:32.567105    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:06:32.568144 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:32.568123    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:06:32.569096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:32.569079    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:06:32.580668 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:32.580644    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:06:32.582296 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:06:32.582278    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:06:44.273573 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:06:44.273532    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:07:14.273189 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:07:14.273152    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:07:41.947856 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:41.947823    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"]
Nov 02 15:07:41.951575 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:41.951556    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"
Nov 02 15:07:42.006988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:42.006954    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86p4v\" (UniqueName: \"kubernetes.io/projected/08e8a676-324b-450f-85b2-8ec2df940dae-kube-api-access-86p4v\") pod \"payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26\" (UID: \"08e8a676-324b-450f-85b2-8ec2df940dae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"
Nov 02 15:07:42.007183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:42.007026    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e8a676-324b-450f-85b2-8ec2df940dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26\" (UID: \"08e8a676-324b-450f-85b2-8ec2df940dae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"
Nov 02 15:07:42.010252 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:42.010221    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"]
Nov 02 15:07:42.056485 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:42.056450    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-c18e79\""
Nov 02 15:07:42.056485 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:42.056449    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-c18e79-dockercfg-ptj9h\""
Nov 02 15:07:42.108670 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:42.108632    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86p4v\" (UniqueName: \"kubernetes.io/projected/08e8a676-324b-450f-85b2-8ec2df940dae-kube-api-access-86p4v\") pod \"payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26\" (UID: \"08e8a676-324b-450f-85b2-8ec2df940dae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"
Nov 02 15:07:42.108849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:42.108702    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e8a676-324b-450f-85b2-8ec2df940dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26\" (UID: \"08e8a676-324b-450f-85b2-8ec2df940dae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"
Nov 02 15:07:42.109507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:42.109482    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e8a676-324b-450f-85b2-8ec2df940dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26\" (UID: \"08e8a676-324b-450f-85b2-8ec2df940dae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"
Nov 02 15:07:42.174518 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:42.174490    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86p4v\" (UniqueName: \"kubernetes.io/projected/08e8a676-324b-450f-85b2-8ec2df940dae-kube-api-access-86p4v\") pod \"payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26\" (UID: \"08e8a676-324b-450f-85b2-8ec2df940dae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"
Nov 02 15:07:42.264555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:42.264471    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"
Nov 02 15:07:42.637165 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:42.637135    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"]
Nov 02 15:07:42.639359 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:07:42.639334    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e8a676_324b_450f_85b2_8ec2df940dae.slice/crio-fe3fb7fa91734949c0b7480b4e6ccd0d739ca27771d3bacb779e48cd6db35ab0 WatchSource:0}: Error finding container fe3fb7fa91734949c0b7480b4e6ccd0d739ca27771d3bacb779e48cd6db35ab0: Status 404 returned error can't find the container with id fe3fb7fa91734949c0b7480b4e6ccd0d739ca27771d3bacb779e48cd6db35ab0
Nov 02 15:07:42.641304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:42.641279    4891 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:07:43.279964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:43.279917    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26" event={"ID":"08e8a676-324b-450f-85b2-8ec2df940dae","Type":"ContainerStarted","Data":"fe3fb7fa91734949c0b7480b4e6ccd0d739ca27771d3bacb779e48cd6db35ab0"}
Nov 02 15:07:44.279851 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:07:44.279814    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:07:46.297691 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:46.297648    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26" event={"ID":"08e8a676-324b-450f-85b2-8ec2df940dae","Type":"ContainerStarted","Data":"158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88"}
Nov 02 15:07:46.298079 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:46.297735    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"
Nov 02 15:07:46.415921 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:46.415865    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26" podStartSLOduration=2.4431494750000002 podStartE2EDuration="5.415849368s" totalImagesPullingTime="2.972699893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:46.41396688 +0000 UTC m=+674.006583680" watchObservedRunningTime="2026-11-02 15:07:46.415849368 +0000 UTC m=+674.008466167"
Nov 02 15:07:47.009888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.009855    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"]
Nov 02 15:07:47.013205 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.013178    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"
Nov 02 15:07:47.051780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.051745    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b5bb68-8249-4e62-a49e-8b85e4a706f1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc\" (UID: \"90b5bb68-8249-4e62-a49e-8b85e4a706f1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"
Nov 02 15:07:47.051959 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.051792    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlq6q\" (UniqueName: \"kubernetes.io/projected/90b5bb68-8249-4e62-a49e-8b85e4a706f1-kube-api-access-wlq6q\") pod \"payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc\" (UID: \"90b5bb68-8249-4e62-a49e-8b85e4a706f1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"
Nov 02 15:07:47.073403 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.073378    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-6dc69b-dockercfg-p7ppf\""
Nov 02 15:07:47.119572 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.119543    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-6dc69b\""
Nov 02 15:07:47.122028 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.121996    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"]
Nov 02 15:07:47.153040 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.153007    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b5bb68-8249-4e62-a49e-8b85e4a706f1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc\" (UID: \"90b5bb68-8249-4e62-a49e-8b85e4a706f1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"
Nov 02 15:07:47.153040 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.153042    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlq6q\" (UniqueName: \"kubernetes.io/projected/90b5bb68-8249-4e62-a49e-8b85e4a706f1-kube-api-access-wlq6q\") pod \"payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc\" (UID: \"90b5bb68-8249-4e62-a49e-8b85e4a706f1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"
Nov 02 15:07:47.153718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.153699    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b5bb68-8249-4e62-a49e-8b85e4a706f1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc\" (UID: \"90b5bb68-8249-4e62-a49e-8b85e4a706f1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"
Nov 02 15:07:47.216000 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.215970    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlq6q\" (UniqueName: \"kubernetes.io/projected/90b5bb68-8249-4e62-a49e-8b85e4a706f1-kube-api-access-wlq6q\") pod \"payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc\" (UID: \"90b5bb68-8249-4e62-a49e-8b85e4a706f1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"
Nov 02 15:07:47.325933 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.325896    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"
Nov 02 15:07:47.697598 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:47.697575    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"]
Nov 02 15:07:47.700400 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:07:47.700351    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b5bb68_8249_4e62_a49e_8b85e4a706f1.slice/crio-f70d97361bdf98d5b0dfca428772e05dd94a45a52e06da6a5c8e3ab08dbcdcce WatchSource:0}: Error finding container f70d97361bdf98d5b0dfca428772e05dd94a45a52e06da6a5c8e3ab08dbcdcce: Status 404 returned error can't find the container with id f70d97361bdf98d5b0dfca428772e05dd94a45a52e06da6a5c8e3ab08dbcdcce
Nov 02 15:07:48.309578 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:48.309539    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc" event={"ID":"90b5bb68-8249-4e62-a49e-8b85e4a706f1","Type":"ContainerStarted","Data":"e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae"}
Nov 02 15:07:48.309771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:48.309601    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc" event={"ID":"90b5bb68-8249-4e62-a49e-8b85e4a706f1","Type":"ContainerStarted","Data":"f70d97361bdf98d5b0dfca428772e05dd94a45a52e06da6a5c8e3ab08dbcdcce"}
Nov 02 15:07:48.309771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:48.309642    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"
Nov 02 15:07:48.426140 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:48.426083    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc" podStartSLOduration=2.426068437 podStartE2EDuration="2.426068437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:48.42481525 +0000 UTC m=+676.017432049" watchObservedRunningTime="2026-11-02 15:07:48.426068437 +0000 UTC m=+676.018685236"
Nov 02 15:07:57.305656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:57.305627    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"
Nov 02 15:07:59.317759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:07:59.317732    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"
Nov 02 15:08:11.301029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.300995    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"]
Nov 02 15:08:11.304849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.304824    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.380996 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.380921    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-db3931\"/\"kube-root-ca.crt\""
Nov 02 15:08:11.381195 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.381160    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-model-cache\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.381195 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.381206    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tls-certs\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.381437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.381373    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-home\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.381437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.381416    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-dshm\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.381555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.381487    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz9zj\" (UniqueName: \"kubernetes.io/projected/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-kube-api-access-pz9zj\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.381664 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.381641    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tmp-dir\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.383773 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.383754    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-db3931\"/\"openshift-service-ca.crt\""
Nov 02 15:08:11.433258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.433219    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"]
Nov 02 15:08:11.433475 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.433458    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-db3931\"/\"simulator-w5-db3931-kserve-self-signed-certs\""
Nov 02 15:08:11.435534 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.435514    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-db3931\"/\"default-dockercfg-7c6qg\""
Nov 02 15:08:11.483048 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.483015    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9zj\" (UniqueName: \"kubernetes.io/projected/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-kube-api-access-pz9zj\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.483048 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.483053    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tmp-dir\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.483260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.483114    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-model-cache\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.483260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.483132    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tls-certs\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.483260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.483173    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-home\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.483260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.483225    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-dshm\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.483661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.483632    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tmp-dir\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.483707 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.483648    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-model-cache\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.483754 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.483712    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-home\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.485639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.485622    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-dshm\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.485933 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.485915    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tls-certs\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.543888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.543860    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9zj\" (UniqueName: \"kubernetes.io/projected/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-kube-api-access-pz9zj\") pod \"simulator-w5-db3931-kserve-74c588ff66-n5h7b\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") " pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:11.618060 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:11.617975    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:12.016122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:12.016096    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"]
Nov 02 15:08:12.017416 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:08:12.017392    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ed60c0c_4ce2_42b3_b8a7_b2b0f2fa2cb4.slice/crio-4bd812c551323d47875a8f16c3a5f8f2b2b47871eb527631c607efac510a8b31 WatchSource:0}: Error finding container 4bd812c551323d47875a8f16c3a5f8f2b2b47871eb527631c607efac510a8b31: Status 404 returned error can't find the container with id 4bd812c551323d47875a8f16c3a5f8f2b2b47871eb527631c607efac510a8b31
Nov 02 15:08:12.448647 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:12.448610    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" event={"ID":"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4","Type":"ContainerStarted","Data":"4bd812c551323d47875a8f16c3a5f8f2b2b47871eb527631c607efac510a8b31"}
Nov 02 15:08:14.273221 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:08:14.273180    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:08:14.463131 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:14.463094    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" event={"ID":"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4","Type":"ContainerStarted","Data":"ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2"}
Nov 02 15:08:14.580041 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:14.579931    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podStartSLOduration=1.842872987 podStartE2EDuration="3.579915967s" totalImagesPullingTime="1.73704298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:14.579200979 +0000 UTC m=+702.171817773" watchObservedRunningTime="2026-11-02 15:08:14.579915967 +0000 UTC m=+702.172532793"
Nov 02 15:08:14.618091 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:14.618054    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:21.618496 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:21.618456    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:21.631246 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:21.631217    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:22.520468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:22.520440    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:08:35.627702 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.627667    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"]
Nov 02 15:08:35.631051 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.631033    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.722264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.720431    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tmp-dir\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.722264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.720489    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-model-cache\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.722264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.720533    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-dshm\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.722264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.720586    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-home\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.722264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.720626    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zc7v\" (UniqueName: \"kubernetes.io/projected/9e14887d-6b64-456e-ad0b-3e0b34789cbe-kube-api-access-8zc7v\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.722264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.720697    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tls-certs\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.732443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.732406    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ba49d0\"/\"openshift-service-ca.crt\""
Nov 02 15:08:35.732637 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.732486    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ba49d0\"/\"simulator-w0-ba49d0-kserve-self-signed-certs\""
Nov 02 15:08:35.732704 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.732665    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ba49d0\"/\"kube-root-ca.crt\""
Nov 02 15:08:35.732831 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.732787    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ba49d0\"/\"default-dockercfg-ksn49\""
Nov 02 15:08:35.735168 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.735144    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"]
Nov 02 15:08:35.821510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.821475    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tmp-dir\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.821510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.821514    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-model-cache\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.821769 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.821573    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-dshm\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.821769 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.821619    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-home\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.821769 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.821648    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zc7v\" (UniqueName: \"kubernetes.io/projected/9e14887d-6b64-456e-ad0b-3e0b34789cbe-kube-api-access-8zc7v\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.821769 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.821711    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tls-certs\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.821989 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.821956    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-model-cache\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.822048 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.822007    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tmp-dir\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.822277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.822252    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-home\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.824268 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.824245    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-dshm\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.824374 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.824296    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tls-certs\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.882654 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.882589    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zc7v\" (UniqueName: \"kubernetes.io/projected/9e14887d-6b64-456e-ad0b-3e0b34789cbe-kube-api-access-8zc7v\") pod \"simulator-w0-ba49d0-kserve-647dc699b9-s9mc5\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") " pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:35.943790 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:35.943744    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:36.341420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:36.341395    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"]
Nov 02 15:08:36.343279 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:08:36.343248    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e14887d_6b64_456e_ad0b_3e0b34789cbe.slice/crio-fb83516a0724a426a99fd4dfbbe98298226a54999750879b81cdd31ca88c1b93 WatchSource:0}: Error finding container fb83516a0724a426a99fd4dfbbe98298226a54999750879b81cdd31ca88c1b93: Status 404 returned error can't find the container with id fb83516a0724a426a99fd4dfbbe98298226a54999750879b81cdd31ca88c1b93
Nov 02 15:08:36.585570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:36.585537    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" event={"ID":"9e14887d-6b64-456e-ad0b-3e0b34789cbe","Type":"ContainerStarted","Data":"dc8c26b7bb9d3458a60f230790d4bbf37e349c7f763e2ab8cc3b73600dbe2460"}
Nov 02 15:08:36.585737 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:36.585584    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" event={"ID":"9e14887d-6b64-456e-ad0b-3e0b34789cbe","Type":"ContainerStarted","Data":"fb83516a0724a426a99fd4dfbbe98298226a54999750879b81cdd31ca88c1b93"}
Nov 02 15:08:36.733802 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:36.733702    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podStartSLOduration=1.733688392 podStartE2EDuration="1.733688392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:36.732155145 +0000 UTC m=+724.324771963" watchObservedRunningTime="2026-11-02 15:08:36.733688392 +0000 UTC m=+724.326305191"
Nov 02 15:08:36.944513 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:36.944478    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:41.741555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.741518    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"]
Nov 02 15:08:41.745220 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.745201    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.797285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.797257    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-42555d\"/\"openshift-service-ca.crt\""
Nov 02 15:08:41.846718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.846684    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-42555d\"/\"default-dockercfg-96rjb\""
Nov 02 15:08:41.846904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.846761    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-42555d\"/\"kube-root-ca.crt\""
Nov 02 15:08:41.846904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.846810    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-42555d\"/\"simulator-w3-42555d-kserve-self-signed-certs\""
Nov 02 15:08:41.849359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.849335    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"]
Nov 02 15:08:41.878985 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.878951    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-home\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.879123 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.879007    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-model-cache\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.879175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.879111    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xpcv\" (UniqueName: \"kubernetes.io/projected/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-kube-api-access-4xpcv\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.879175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.879150    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tmp-dir\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.879253 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.879234    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-dshm\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.879288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.879269    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tls-certs\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.980494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.980449    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-model-cache\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.980699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.980515    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpcv\" (UniqueName: \"kubernetes.io/projected/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-kube-api-access-4xpcv\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.980699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.980542    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tmp-dir\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.980699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.980576    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-dshm\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.980699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.980591    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tls-certs\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.980699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.980615    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-home\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.980980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.980948    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-model-cache\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.980980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.980969    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tmp-dir\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.981086 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.981036    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-home\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.982859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.982835    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-dshm\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:41.983268 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:41.983246    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tls-certs\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:42.042257 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:42.042218    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpcv\" (UniqueName: \"kubernetes.io/projected/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-kube-api-access-4xpcv\") pod \"simulator-w3-42555d-kserve-846b876d89-zl28f\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") " pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:42.057349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:42.057306    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:42.447409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:42.447382    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"]
Nov 02 15:08:42.449000 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:08:42.448979    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c42e0c6_3b38_4ef8_9278_1a3b0b8432e1.slice/crio-2c407b91d77c8c9877e138de58b092f3580e7eb29c5e82b8bb6e0387f6ab594f WatchSource:0}: Error finding container 2c407b91d77c8c9877e138de58b092f3580e7eb29c5e82b8bb6e0387f6ab594f: Status 404 returned error can't find the container with id 2c407b91d77c8c9877e138de58b092f3580e7eb29c5e82b8bb6e0387f6ab594f
Nov 02 15:08:42.625729 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:42.625691    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" event={"ID":"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1","Type":"ContainerStarted","Data":"f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba"}
Nov 02 15:08:42.625729 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:42.625734    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" event={"ID":"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1","Type":"ContainerStarted","Data":"2c407b91d77c8c9877e138de58b092f3580e7eb29c5e82b8bb6e0387f6ab594f"}
Nov 02 15:08:42.743247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:42.743139    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podStartSLOduration=1.7431247079999999 podStartE2EDuration="1.743124708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:42.742115616 +0000 UTC m=+730.334732418" watchObservedRunningTime="2026-11-02 15:08:42.743124708 +0000 UTC m=+730.335741518"
Nov 02 15:08:43.057745 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:43.057704    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:44.272932 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:08:44.272891    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:08:45.944288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:45.944246    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:45.957128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:45.957098    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:46.664946 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:46.664919    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:08:52.057461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:52.057419    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:52.070361 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:52.070335    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:08:52.700295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:08:52.700266    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:09:00.194913 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.194878    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"]
Nov 02 15:09:00.198517 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.198499    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.250860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.250832    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-de35cd\"/\"openshift-service-ca.crt\""
Nov 02 15:09:00.263507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.263480    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-tmp-dir\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.263687 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.263601    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0edd53-e43a-42e6-884a-92620f077b34-tls-certs\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.263687 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.263644    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-home\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.263816 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.263730    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h89n9\" (UniqueName: \"kubernetes.io/projected/ca0edd53-e43a-42e6-884a-92620f077b34-kube-api-access-h89n9\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.263816 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.263769    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-dshm\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.263816 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.263799    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-model-cache\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.300237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.300179    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-de35cd\"/\"simulator-w2-de35cd-kserve-self-signed-certs\""
Nov 02 15:09:00.300237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.300179    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-de35cd\"/\"default-dockercfg-w8wnr\""
Nov 02 15:09:00.300451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.300248    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-de35cd\"/\"kube-root-ca.crt\""
Nov 02 15:09:00.303556 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.303534    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"]
Nov 02 15:09:00.364501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.364465    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0edd53-e43a-42e6-884a-92620f077b34-tls-certs\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.364501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.364504    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-home\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.364712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.364524    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h89n9\" (UniqueName: \"kubernetes.io/projected/ca0edd53-e43a-42e6-884a-92620f077b34-kube-api-access-h89n9\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.364712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.364657    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-dshm\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.364712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.364699    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-model-cache\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.364827 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.364804    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-tmp-dir\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.365149 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.365128    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-home\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.365149 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.365139    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-model-cache\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.365283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.365169    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-tmp-dir\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.366785 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.366769    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-dshm\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.367064 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.367048    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0edd53-e43a-42e6-884a-92620f077b34-tls-certs\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.425296 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.425262    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h89n9\" (UniqueName: \"kubernetes.io/projected/ca0edd53-e43a-42e6-884a-92620f077b34-kube-api-access-h89n9\") pod \"simulator-w2-de35cd-kserve-67cc89c4fc-zsnll\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") " pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.512040 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.511959    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:00.900545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:00.900517    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"]
Nov 02 15:09:00.902307 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:09:00.902277    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca0edd53_e43a_42e6_884a_92620f077b34.slice/crio-16d4daaef7396f583169e4e5192b1c0b696cc70984181d8765e09deda7eba5e6 WatchSource:0}: Error finding container 16d4daaef7396f583169e4e5192b1c0b696cc70984181d8765e09deda7eba5e6: Status 404 returned error can't find the container with id 16d4daaef7396f583169e4e5192b1c0b696cc70984181d8765e09deda7eba5e6
Nov 02 15:09:01.741383 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:01.741344    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" event={"ID":"ca0edd53-e43a-42e6-884a-92620f077b34","Type":"ContainerStarted","Data":"f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd"}
Nov 02 15:09:01.741876 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:01.741393    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" event={"ID":"ca0edd53-e43a-42e6-884a-92620f077b34","Type":"ContainerStarted","Data":"16d4daaef7396f583169e4e5192b1c0b696cc70984181d8765e09deda7eba5e6"}
Nov 02 15:09:01.868172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:01.868112    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podStartSLOduration=1.868096762 podStartE2EDuration="1.868096762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:01.864413415 +0000 UTC m=+749.457030211" watchObservedRunningTime="2026-11-02 15:09:01.868096762 +0000 UTC m=+749.460713565"
Nov 02 15:09:02.512469 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:02.512426    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:10.512725 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:10.512687    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:10.525854 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:10.525829    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:10.809797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:10.809770    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:09:12.699775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.699682    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"]
Nov 02 15:09:12.703564 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.703542    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.756168 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.756138    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-019efe\"/\"kube-root-ca.crt\""
Nov 02 15:09:12.759355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.759178    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-019efe\"/\"default-dockercfg-2g9cz\""
Nov 02 15:09:12.759355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.759185    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-019efe\"/\"openshift-service-ca.crt\""
Nov 02 15:09:12.759510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.759364    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-019efe\"/\"simulator-w1-019efe-kserve-self-signed-certs\""
Nov 02 15:09:12.771423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.771396    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-model-cache\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.771542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.771445    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7nd7\" (UniqueName: \"kubernetes.io/projected/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-kube-api-access-c7nd7\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.771594 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.771547    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tls-certs\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.771594 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.771588    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-dshm\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.771678 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.771650    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-home\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.771720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.771691    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tmp-dir\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.809148 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.809114    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"]
Nov 02 15:09:12.874348 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.874272    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7nd7\" (UniqueName: \"kubernetes.io/projected/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-kube-api-access-c7nd7\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.874595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.874371    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tls-certs\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.874595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.874426    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-dshm\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.874595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.874468    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-home\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.874595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.874500    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tmp-dir\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.874595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.874557    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-model-cache\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.875028 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.875004    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-model-cache\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.878204 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.878179    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tmp-dir\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.878653 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.878625    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-home\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.878976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.878953    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tls-certs\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.880032 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.880012    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-dshm\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:12.936301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:12.936264    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7nd7\" (UniqueName: \"kubernetes.io/projected/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-kube-api-access-c7nd7\") pod \"simulator-w1-019efe-kserve-56f7ff8c88-r9kvv\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") " pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:13.015528 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:13.015433    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:13.402050 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:13.402009    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"]
Nov 02 15:09:13.404158 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:09:13.404132    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd4f9a33_e005_4bd4_97a4_e65d83ef4220.slice/crio-7023af2f7b7a0f3f7171e563f41ec8dea6a814a43b986b391084f848d4c6cc02 WatchSource:0}: Error finding container 7023af2f7b7a0f3f7171e563f41ec8dea6a814a43b986b391084f848d4c6cc02: Status 404 returned error can't find the container with id 7023af2f7b7a0f3f7171e563f41ec8dea6a814a43b986b391084f848d4c6cc02
Nov 02 15:09:13.818926 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:13.818893    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" event={"ID":"fd4f9a33-e005-4bd4-97a4-e65d83ef4220","Type":"ContainerStarted","Data":"64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e"}
Nov 02 15:09:13.819331 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:13.818941    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" event={"ID":"fd4f9a33-e005-4bd4-97a4-e65d83ef4220","Type":"ContainerStarted","Data":"7023af2f7b7a0f3f7171e563f41ec8dea6a814a43b986b391084f848d4c6cc02"}
Nov 02 15:09:13.936742 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:13.936687    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podStartSLOduration=1.936671969 podStartE2EDuration="1.936671969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:13.934765581 +0000 UTC m=+761.527382380" watchObservedRunningTime="2026-11-02 15:09:13.936671969 +0000 UTC m=+761.529288779"
Nov 02 15:09:14.015829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:14.015799    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:14.272734 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:09:14.272691    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:09:16.644061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.644028    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"]
Nov 02 15:09:16.647429 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.647412    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.712141 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.712111    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-home\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.712293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.712243    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kwf\" (UniqueName: \"kubernetes.io/projected/ac326ed5-2b28-47d6-8ac2-216f41a29e45-kube-api-access-d9kwf\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.712380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.712323    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tls-certs\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.712453 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.712431    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-model-cache\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.712512 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.712482    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-dshm\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.712575 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.712556    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tmp-dir\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.749186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.749156    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-db3931\"/\"premium-w5-db3931-kserve-self-signed-certs\""
Nov 02 15:09:16.752050 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.752029    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"]
Nov 02 15:09:16.813116 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.813083    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-dshm\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.813294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.813144    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tmp-dir\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.813294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.813204    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-home\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.813294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.813275    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kwf\" (UniqueName: \"kubernetes.io/projected/ac326ed5-2b28-47d6-8ac2-216f41a29e45-kube-api-access-d9kwf\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.813432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.813343    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tls-certs\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.813432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.813391    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-model-cache\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.813661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.813632    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tmp-dir\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.813792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.813640    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-home\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.813792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.813716    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-model-cache\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.815508 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.815481    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-dshm\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.815849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.815828    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tls-certs\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.893841 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.893807    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kwf\" (UniqueName: \"kubernetes.io/projected/ac326ed5-2b28-47d6-8ac2-216f41a29e45-kube-api-access-d9kwf\") pod \"premium-w5-db3931-kserve-5fb7f878c5-dczxm\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") " pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:16.959956 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:16.959854    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:17.379706 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:17.379672    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"]
Nov 02 15:09:17.381405 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:09:17.381378    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac326ed5_2b28_47d6_8ac2_216f41a29e45.slice/crio-5f55184bd4249941070b055e9d5abbc54ade3d323e640cea9bf3f58d6c807715 WatchSource:0}: Error finding container 5f55184bd4249941070b055e9d5abbc54ade3d323e640cea9bf3f58d6c807715: Status 404 returned error can't find the container with id 5f55184bd4249941070b055e9d5abbc54ade3d323e640cea9bf3f58d6c807715
Nov 02 15:09:17.845971 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:17.845931    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" event={"ID":"ac326ed5-2b28-47d6-8ac2-216f41a29e45","Type":"ContainerStarted","Data":"3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710"}
Nov 02 15:09:17.846399 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:17.845979    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" event={"ID":"ac326ed5-2b28-47d6-8ac2-216f41a29e45","Type":"ContainerStarted","Data":"5f55184bd4249941070b055e9d5abbc54ade3d323e640cea9bf3f58d6c807715"}
Nov 02 15:09:17.960739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:17.960707    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:17.964695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:17.964650    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podStartSLOduration=1.964637465 podStartE2EDuration="1.964637465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:17.96217102 +0000 UTC m=+765.554787820" watchObservedRunningTime="2026-11-02 15:09:17.964637465 +0000 UTC m=+765.557254274"
Nov 02 15:09:23.015551 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:23.015513    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:23.028529 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:23.028502    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:23.890948 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:23.890918    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:09:26.960369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:26.960334    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:26.972981 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:26.972951    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:27.914558 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:27.914530    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:09:28.711396 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.711361    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"]
Nov 02 15:09:28.715083 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.715068    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.819297 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.819262    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6dc69b\"/\"kube-root-ca.crt\""
Nov 02 15:09:28.819506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.819377    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6dc69b\"/\"openshift-service-ca.crt\""
Nov 02 15:09:28.819506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.819471    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6dc69b\"/\"simulator-w4-6dc69b-kserve-self-signed-certs\""
Nov 02 15:09:28.819506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.819484    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6dc69b\"/\"default-dockercfg-j445d\""
Nov 02 15:09:28.822214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.822184    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"]
Nov 02 15:09:28.827074 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.827052    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tmp-dir\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.827170 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.827084    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-home\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.827212 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.827197    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tls-certs\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.827304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.827221    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xjb7\" (UniqueName: \"kubernetes.io/projected/07f03b2d-75da-4e3a-8e75-5608d55a27f8-kube-api-access-2xjb7\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.827391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.827371    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-dshm\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.827461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.827445    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-model-cache\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.928723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.928689    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tls-certs\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.928723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.928723    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjb7\" (UniqueName: \"kubernetes.io/projected/07f03b2d-75da-4e3a-8e75-5608d55a27f8-kube-api-access-2xjb7\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.928980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.928774    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-dshm\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.928980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.928816    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-model-cache\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.928980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.928860    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tmp-dir\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.928980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.928881    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-home\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.929282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.929259    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tmp-dir\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.929362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.929271    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-home\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.929431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.929355    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-model-cache\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.931262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.931233    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-dshm\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.931542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.931522    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tls-certs\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:28.990288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:28.990205    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjb7\" (UniqueName: \"kubernetes.io/projected/07f03b2d-75da-4e3a-8e75-5608d55a27f8-kube-api-access-2xjb7\") pod \"simulator-w4-6dc69b-kserve-75d48945f-fm4qz\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") " pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:29.026949 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:29.026906    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:29.412736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:29.412698    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"]
Nov 02 15:09:29.465423 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:09:29.465394    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07f03b2d_75da_4e3a_8e75_5608d55a27f8.slice/crio-ac407d29b76bca3a96dad63892c57fb146c51a409c9ff73091a11ca8cf31b08e WatchSource:0}: Error finding container ac407d29b76bca3a96dad63892c57fb146c51a409c9ff73091a11ca8cf31b08e: Status 404 returned error can't find the container with id ac407d29b76bca3a96dad63892c57fb146c51a409c9ff73091a11ca8cf31b08e
Nov 02 15:09:29.918624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:29.918589    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" event={"ID":"07f03b2d-75da-4e3a-8e75-5608d55a27f8","Type":"ContainerStarted","Data":"6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31"}
Nov 02 15:09:29.919026 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:29.918636    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" event={"ID":"07f03b2d-75da-4e3a-8e75-5608d55a27f8","Type":"ContainerStarted","Data":"ac407d29b76bca3a96dad63892c57fb146c51a409c9ff73091a11ca8cf31b08e"}
Nov 02 15:09:30.027784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:30.027740    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:30.037848 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:30.037805    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podStartSLOduration=2.037791953 podStartE2EDuration="2.037791953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:30.036001329 +0000 UTC m=+777.628618128" watchObservedRunningTime="2026-11-02 15:09:30.037791953 +0000 UTC m=+777.630408752"
Nov 02 15:09:39.028125 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:39.028078    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:39.040748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:39.040723    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:39.990043 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:39.990014    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:09:44.273941 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:09:44.273903    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:09:45.212534 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.212491    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"]
Nov 02 15:09:45.217292 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.217276    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.320999 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.320967    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c18e79\"/\"openshift-service-ca.crt\""
Nov 02 15:09:45.321444 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.320967    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c18e79\"/\"kube-root-ca.crt\""
Nov 02 15:09:45.321444 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.321127    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c18e79\"/\"default-dockercfg-nmwlk\""
Nov 02 15:09:45.321444 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.321194    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c18e79\"/\"simulator-w6-c18e79-kserve-self-signed-certs\""
Nov 02 15:09:45.327145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.327113    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"]
Nov 02 15:09:45.394772 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.394730    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-dshm\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.394962 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.394790    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-home\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.394962 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.394848    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-model-cache\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.394962 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.394882    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhxl\" (UniqueName: \"kubernetes.io/projected/2dbba973-9a21-4bed-9112-e9d0f9282d68-kube-api-access-xbhxl\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.394962 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.394933    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-tmp-dir\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.395107 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.395048    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbba973-9a21-4bed-9112-e9d0f9282d68-tls-certs\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.496024 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.495910    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-model-cache\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.496024 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.495995    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhxl\" (UniqueName: \"kubernetes.io/projected/2dbba973-9a21-4bed-9112-e9d0f9282d68-kube-api-access-xbhxl\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.496263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.496073    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-tmp-dir\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.496263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.496135    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbba973-9a21-4bed-9112-e9d0f9282d68-tls-certs\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.496263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.496173    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-dshm\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.496263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.496220    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-home\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.496520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.496433    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-model-cache\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.496520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.496477    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-tmp-dir\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.496637 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.496609    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-home\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.498595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.498564    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-dshm\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.498914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.498889    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbba973-9a21-4bed-9112-e9d0f9282d68-tls-certs\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.556860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.556827    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhxl\" (UniqueName: \"kubernetes.io/projected/2dbba973-9a21-4bed-9112-e9d0f9282d68-kube-api-access-xbhxl\") pod \"simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") " pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:45.829129 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:45.829096    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:46.206561 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:46.206475    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"]
Nov 02 15:09:46.247429 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:09:46.247398    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dbba973_9a21_4bed_9112_e9d0f9282d68.slice/crio-39ce053f951f990510ea29351abd272375d578fd26020905bd0df37f5ae884b4 WatchSource:0}: Error finding container 39ce053f951f990510ea29351abd272375d578fd26020905bd0df37f5ae884b4: Status 404 returned error can't find the container with id 39ce053f951f990510ea29351abd272375d578fd26020905bd0df37f5ae884b4
Nov 02 15:09:47.022478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:47.022436    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" event={"ID":"2dbba973-9a21-4bed-9112-e9d0f9282d68","Type":"ContainerStarted","Data":"c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8"}
Nov 02 15:09:47.022938 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:47.022487    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" event={"ID":"2dbba973-9a21-4bed-9112-e9d0f9282d68","Type":"ContainerStarted","Data":"39ce053f951f990510ea29351abd272375d578fd26020905bd0df37f5ae884b4"}
Nov 02 15:09:47.141090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:47.141040    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podStartSLOduration=2.141025251 podStartE2EDuration="2.141025251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:47.139089284 +0000 UTC m=+794.731706082" watchObservedRunningTime="2026-11-02 15:09:47.141025251 +0000 UTC m=+794.733642050"
Nov 02 15:09:47.830190 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:47.830154    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:55.829557 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:55.829522    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:55.842095 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:55.842073    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:09:56.086287 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:09:56.086212    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:10:10.625096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.625062    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"]
Nov 02 15:10:10.629029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.629007    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.702323 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.702283    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"]
Nov 02 15:10:10.718138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.718108    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-home\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.718138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.718137    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91550145-6eaf-4595-83fa-868563047896-tls-certs\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.718350 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.718156    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-dshm\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.718350 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.718219    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pjg7\" (UniqueName: \"kubernetes.io/projected/91550145-6eaf-4595-83fa-868563047896-kube-api-access-8pjg7\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.718434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.718365    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-model-cache\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.718484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.718471    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-tmp-dir\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.733233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.733212    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-de35cd\"/\"premium-w2-de35cd-kserve-self-signed-certs\""
Nov 02 15:10:10.819698 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.819653    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-home\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.819698 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.819697    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91550145-6eaf-4595-83fa-868563047896-tls-certs\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.819914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.819721    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-dshm\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.819914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.819743    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pjg7\" (UniqueName: \"kubernetes.io/projected/91550145-6eaf-4595-83fa-868563047896-kube-api-access-8pjg7\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.819914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.819809    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-model-cache\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.820046 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.819932    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-tmp-dir\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.820165 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.820144    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-home\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.820249 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.820215    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-model-cache\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.820289 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.820254    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-tmp-dir\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.821994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.821966    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-dshm\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.822440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.822423    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91550145-6eaf-4595-83fa-868563047896-tls-certs\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.881372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.881271    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pjg7\" (UniqueName: \"kubernetes.io/projected/91550145-6eaf-4595-83fa-868563047896-kube-api-access-8pjg7\") pod \"premium-w2-de35cd-kserve-5bb6478896-v7b9p\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") " pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:10.941760 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:10.941725    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:11.365285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:11.365255    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"]
Nov 02 15:10:11.366738 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:10:11.366708    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91550145_6eaf_4595_83fa_868563047896.slice/crio-c04f6df4ff6f93457709607e267ee26361e0f6a5d6ed57623de7a3a522920f6c WatchSource:0}: Error finding container c04f6df4ff6f93457709607e267ee26361e0f6a5d6ed57623de7a3a522920f6c: Status 404 returned error can't find the container with id c04f6df4ff6f93457709607e267ee26361e0f6a5d6ed57623de7a3a522920f6c
Nov 02 15:10:12.168998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:12.168961    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" event={"ID":"91550145-6eaf-4595-83fa-868563047896","Type":"ContainerStarted","Data":"34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744"}
Nov 02 15:10:12.168998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:12.169004    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" event={"ID":"91550145-6eaf-4595-83fa-868563047896","Type":"ContainerStarted","Data":"c04f6df4ff6f93457709607e267ee26361e0f6a5d6ed57623de7a3a522920f6c"}
Nov 02 15:10:12.286882 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:12.286822    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podStartSLOduration=2.286806645 podStartE2EDuration="2.286806645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:12.285518792 +0000 UTC m=+819.878135593" watchObservedRunningTime="2026-11-02 15:10:12.286806645 +0000 UTC m=+819.879423443"
Nov 02 15:10:12.947435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:12.944402    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:14.273375 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:10:14.273333    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:10:20.942069 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:20.942030    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:20.954872 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:20.954844    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:21.233359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:21.233258    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:10:44.273400 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:10:44.273296    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:10:47.450958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.450929    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"]
Nov 02 15:10:47.454526 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.454509    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:47.511340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.511287    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"]
Nov 02 15:10:47.516562 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.516530    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ca0ac4-54kpg\" (UID: \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:47.516718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.516575    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m64vg\" (UniqueName: \"kubernetes.io/projected/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-kube-api-access-m64vg\") pod \"maas-api-revoke-sub-60ca0ac4-54kpg\" (UID: \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:47.516802 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.516746    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ca0ac4-54kpg\" (UID: \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:47.556296 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.556258    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:10:47.556296 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.556260    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:10:47.556296 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.556268    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:10:47.556541 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.556358    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:10:47.618246 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.618213    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ca0ac4-54kpg\" (UID: \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:47.618463 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.618329    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ca0ac4-54kpg\" (UID: \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:47.618463 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.618395    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m64vg\" (UniqueName: \"kubernetes.io/projected/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-kube-api-access-m64vg\") pod \"maas-api-revoke-sub-60ca0ac4-54kpg\" (UID: \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:47.619137 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.619108    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ca0ac4-54kpg\" (UID: \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:47.620865 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.620846    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ca0ac4-54kpg\" (UID: \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:47.679771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.679724    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m64vg\" (UniqueName: \"kubernetes.io/projected/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-kube-api-access-m64vg\") pod \"maas-api-revoke-sub-60ca0ac4-54kpg\" (UID: \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:47.768052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:47.767972    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:48.145283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:48.145257    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"]
Nov 02 15:10:48.146777 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:10:48.146752    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc01975_8c7b_4518_a5db_ab7cc59e8b15.slice/crio-3aa2551f7906e47545899d304d39f3532efa3f544a1e748912007c731f62ed33 WatchSource:0}: Error finding container 3aa2551f7906e47545899d304d39f3532efa3f544a1e748912007c731f62ed33: Status 404 returned error can't find the container with id 3aa2551f7906e47545899d304d39f3532efa3f544a1e748912007c731f62ed33
Nov 02 15:10:48.389958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:48.389922    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg" event={"ID":"9fc01975-8c7b-4518-a5db-ab7cc59e8b15","Type":"ContainerStarted","Data":"3aa2551f7906e47545899d304d39f3532efa3f544a1e748912007c731f62ed33"}
Nov 02 15:10:53.438226 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:53.438189    4891 generic.go:356] "Generic (PLEG): container finished" podID="9fc01975-8c7b-4518-a5db-ab7cc59e8b15" containerID="beab5fbea63f3c5abe5a78f9b156079921d1b05e2e0e372ecb9ad0f664373523" exitCode=0
Nov 02 15:10:53.438653 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:53.438272    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg" event={"ID":"9fc01975-8c7b-4518-a5db-ab7cc59e8b15","Type":"ContainerDied","Data":"beab5fbea63f3c5abe5a78f9b156079921d1b05e2e0e372ecb9ad0f664373523"}
Nov 02 15:10:54.594804 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:54.594777    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:54.696641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:54.696555    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-openshift-ca\") pod \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\" (UID: \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\") "
Nov 02 15:10:54.696641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:54.696614    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-kube-api-access-m64vg\" (UniqueName: \"kubernetes.io/projected/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-kube-api-access-m64vg\") pod \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\" (UID: \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\") "
Nov 02 15:10:54.696869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:54.696721    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-xks-ca\") pod \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\" (UID: \"9fc01975-8c7b-4518-a5db-ab7cc59e8b15\") "
Nov 02 15:10:54.696951 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:54.696915    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-openshift-ca" pod "9fc01975-8c7b-4518-a5db-ab7cc59e8b15" (UID: "9fc01975-8c7b-4518-a5db-ab7cc59e8b15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:10:54.697125 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:54.697102    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:10:54.698880 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:54.698858    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-xks-ca" pod "9fc01975-8c7b-4518-a5db-ab7cc59e8b15" (UID: "9fc01975-8c7b-4518-a5db-ab7cc59e8b15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:10:54.698960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:54.698859    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-kube-api-access-m64vg" pod "9fc01975-8c7b-4518-a5db-ab7cc59e8b15" (UID: "9fc01975-8c7b-4518-a5db-ab7cc59e8b15"). InnerVolumeSpecName "kube-api-access-m64vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:10:54.797981 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:54.797941    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m64vg\" (UniqueName: \"kubernetes.io/projected/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-kube-api-access-m64vg\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:10:54.797981 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:54.797978    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc01975-8c7b-4518-a5db-ab7cc59e8b15-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:10:55.451391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:55.451357    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"
Nov 02 15:10:55.451391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:55.451371    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg" event={"ID":"9fc01975-8c7b-4518-a5db-ab7cc59e8b15","Type":"ContainerDied","Data":"3aa2551f7906e47545899d304d39f3532efa3f544a1e748912007c731f62ed33"}
Nov 02 15:10:55.451599 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:10:55.451449    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3aa2551f7906e47545899d304d39f3532efa3f544a1e748912007c731f62ed33"
Nov 02 15:11:12.302821 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.302784    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"]
Nov 02 15:11:12.303241 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.303227    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9fc01975-8c7b-4518-a5db-ab7cc59e8b15" containerName="revoke-keys"
Nov 02 15:11:12.303285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.303242    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc01975-8c7b-4518-a5db-ab7cc59e8b15" containerName="revoke-keys"
Nov 02 15:11:12.303344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.303326    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc01975-8c7b-4518-a5db-ab7cc59e8b15" containerName="revoke-keys"
Nov 02 15:11:12.306855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.306827    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.365640 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.365606    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"]
Nov 02 15:11:12.382986 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.382949    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73d2092c-52f2-499d-85f7-8681595ea6b3-tls-certs\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.383172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.383007    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-tmp-dir\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.383172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.383037    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-dshm\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.383172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.383067    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9zf2\" (UniqueName: \"kubernetes.io/projected/73d2092c-52f2-499d-85f7-8681595ea6b3-kube-api-access-x9zf2\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.383172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.383096    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-model-cache\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.383172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.383135    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-home\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.410517 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.410489    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c18e79\"/\"premium-w6-c18e79-kserve-self-signed-certs\""
Nov 02 15:11:12.483842 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.483806    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-tmp-dir\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.484029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.483860    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-dshm\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.484029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.483894    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9zf2\" (UniqueName: \"kubernetes.io/projected/73d2092c-52f2-499d-85f7-8681595ea6b3-kube-api-access-x9zf2\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.484029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.483917    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-model-cache\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.484185 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.484048    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-home\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.484185 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.484136    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73d2092c-52f2-499d-85f7-8681595ea6b3-tls-certs\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.484297 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.484222    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-tmp-dir\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.484369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.484302    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-model-cache\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.484468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.484443    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-home\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.486133 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.486110    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-dshm\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.486659 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.486637    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73d2092c-52f2-499d-85f7-8681595ea6b3-tls-certs\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.552306 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.552272    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9zf2\" (UniqueName: \"kubernetes.io/projected/73d2092c-52f2-499d-85f7-8681595ea6b3-kube-api-access-x9zf2\") pod \"premium-w6-c18e79-kserve-64f6f85cbf-gzfvs\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") " pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:12.619695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:12.619604    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:13.021415 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:13.021388    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"]
Nov 02 15:11:13.022887 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:11:13.022857    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73d2092c_52f2_499d_85f7_8681595ea6b3.slice/crio-5051fcde78f219c549c759121e99746cd25923906e7d4b0979bcd0d9949b2b61 WatchSource:0}: Error finding container 5051fcde78f219c549c759121e99746cd25923906e7d4b0979bcd0d9949b2b61: Status 404 returned error can't find the container with id 5051fcde78f219c549c759121e99746cd25923906e7d4b0979bcd0d9949b2b61
Nov 02 15:11:13.575726 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:13.575687    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" event={"ID":"73d2092c-52f2-499d-85f7-8681595ea6b3","Type":"ContainerStarted","Data":"5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7"}
Nov 02 15:11:13.575726 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:13.575731    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" event={"ID":"73d2092c-52f2-499d-85f7-8681595ea6b3","Type":"ContainerStarted","Data":"5051fcde78f219c549c759121e99746cd25923906e7d4b0979bcd0d9949b2b61"}
Nov 02 15:11:13.620487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:13.620451    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:13.700112 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:13.700057    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podStartSLOduration=1.7000417570000002 podStartE2EDuration="1.700041757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:13.697165249 +0000 UTC m=+881.289791000" watchObservedRunningTime="2026-11-02 15:11:13.700041757 +0000 UTC m=+881.292658555"
Nov 02 15:11:14.273747 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:11:14.273706    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:11:16.475711 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.475672    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"]
Nov 02 15:11:16.479558 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.479537    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:16.532150 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.532114    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:11:16.581699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.581660    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:16.581699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.581668    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:11:16.581945 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.581681    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:11:16.585259 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.585159    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"]
Nov 02 15:11:16.632464 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.632432    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786ebfb1-466pn\" (UID: \"f7cacf64-9854-423d-89ae-6bdae3d68681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:16.632684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.632499    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd58z\" (UniqueName: \"kubernetes.io/projected/f7cacf64-9854-423d-89ae-6bdae3d68681-kube-api-access-xd58z\") pod \"maas-api-revoke-sub-786ebfb1-466pn\" (UID: \"f7cacf64-9854-423d-89ae-6bdae3d68681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:16.632750 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.632705    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786ebfb1-466pn\" (UID: \"f7cacf64-9854-423d-89ae-6bdae3d68681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:16.733529 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.733443    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786ebfb1-466pn\" (UID: \"f7cacf64-9854-423d-89ae-6bdae3d68681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:16.733529 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.733492    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd58z\" (UniqueName: \"kubernetes.io/projected/f7cacf64-9854-423d-89ae-6bdae3d68681-kube-api-access-xd58z\") pod \"maas-api-revoke-sub-786ebfb1-466pn\" (UID: \"f7cacf64-9854-423d-89ae-6bdae3d68681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:16.733759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.733543    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786ebfb1-466pn\" (UID: \"f7cacf64-9854-423d-89ae-6bdae3d68681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:16.734089 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.734068    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786ebfb1-466pn\" (UID: \"f7cacf64-9854-423d-89ae-6bdae3d68681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:16.735841 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.735808    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786ebfb1-466pn\" (UID: \"f7cacf64-9854-423d-89ae-6bdae3d68681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:16.794538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:16.794506    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd58z\" (UniqueName: \"kubernetes.io/projected/f7cacf64-9854-423d-89ae-6bdae3d68681-kube-api-access-xd58z\") pod \"maas-api-revoke-sub-786ebfb1-466pn\" (UID: \"f7cacf64-9854-423d-89ae-6bdae3d68681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:17.093605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:17.093573    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:17.482133 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:17.482036    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"]
Nov 02 15:11:17.521509 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:11:17.521474    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7cacf64_9854_423d_89ae_6bdae3d68681.slice/crio-0558d657f589e974f44c1982f9c910ba106b53da88bbe0061cd5553b2549bb09 WatchSource:0}: Error finding container 0558d657f589e974f44c1982f9c910ba106b53da88bbe0061cd5553b2549bb09: Status 404 returned error can't find the container with id 0558d657f589e974f44c1982f9c910ba106b53da88bbe0061cd5553b2549bb09
Nov 02 15:11:17.611599 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:17.611556    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn" event={"ID":"f7cacf64-9854-423d-89ae-6bdae3d68681","Type":"ContainerStarted","Data":"0558d657f589e974f44c1982f9c910ba106b53da88bbe0061cd5553b2549bb09"}
Nov 02 15:11:18.619379 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:18.619343    4891 generic.go:356] "Generic (PLEG): container finished" podID="f7cacf64-9854-423d-89ae-6bdae3d68681" containerID="ac457d8919c163c1fcf71d907b92b755ee057010aa70ddda54f8bcd15cc82c8f" exitCode=0
Nov 02 15:11:18.619864 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:18.619430    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn" event={"ID":"f7cacf64-9854-423d-89ae-6bdae3d68681","Type":"ContainerDied","Data":"ac457d8919c163c1fcf71d907b92b755ee057010aa70ddda54f8bcd15cc82c8f"}
Nov 02 15:11:19.775934 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:19.775909    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:19.865873 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:19.865838    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7cacf64-9854-423d-89ae-6bdae3d68681-kube-api-access-xd58z\" (UniqueName: \"kubernetes.io/projected/f7cacf64-9854-423d-89ae-6bdae3d68681-kube-api-access-xd58z\") pod \"f7cacf64-9854-423d-89ae-6bdae3d68681\" (UID: \"f7cacf64-9854-423d-89ae-6bdae3d68681\") "
Nov 02 15:11:19.866072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:19.866011    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-xks-ca\") pod \"f7cacf64-9854-423d-89ae-6bdae3d68681\" (UID: \"f7cacf64-9854-423d-89ae-6bdae3d68681\") "
Nov 02 15:11:19.866072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:19.866065    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-openshift-ca\") pod \"f7cacf64-9854-423d-89ae-6bdae3d68681\" (UID: \"f7cacf64-9854-423d-89ae-6bdae3d68681\") "
Nov 02 15:11:19.866492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:19.866464    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-openshift-ca" pod "f7cacf64-9854-423d-89ae-6bdae3d68681" (UID: "f7cacf64-9854-423d-89ae-6bdae3d68681"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:19.868018 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:19.867991    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-xks-ca" pod "f7cacf64-9854-423d-89ae-6bdae3d68681" (UID: "f7cacf64-9854-423d-89ae-6bdae3d68681"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:19.868018 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:19.868006    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7cacf64-9854-423d-89ae-6bdae3d68681-kube-api-access-xd58z" pod "f7cacf64-9854-423d-89ae-6bdae3d68681" (UID: "f7cacf64-9854-423d-89ae-6bdae3d68681"). InnerVolumeSpecName "kube-api-access-xd58z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:19.967728 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:19.967627    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd58z\" (UniqueName: \"kubernetes.io/projected/f7cacf64-9854-423d-89ae-6bdae3d68681-kube-api-access-xd58z\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:19.967728 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:19.967662    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:19.967728 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:19.967684    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7cacf64-9854-423d-89ae-6bdae3d68681-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:20.632789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:20.632759    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn" event={"ID":"f7cacf64-9854-423d-89ae-6bdae3d68681","Type":"ContainerDied","Data":"0558d657f589e974f44c1982f9c910ba106b53da88bbe0061cd5553b2549bb09"}
Nov 02 15:11:20.632960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:20.632806    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0558d657f589e974f44c1982f9c910ba106b53da88bbe0061cd5553b2549bb09"
Nov 02 15:11:20.632960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:20.632772    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"
Nov 02 15:11:22.620492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:22.620452    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:22.633363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:22.633332    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:22.665939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:22.665908    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:11:27.660989 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.660943    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"]
Nov 02 15:11:27.662221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.662190    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f7cacf64-9854-423d-89ae-6bdae3d68681" containerName="revoke-keys"
Nov 02 15:11:27.662397 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.662222    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7cacf64-9854-423d-89ae-6bdae3d68681" containerName="revoke-keys"
Nov 02 15:11:27.662468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.662415    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7cacf64-9854-423d-89ae-6bdae3d68681" containerName="revoke-keys"
Nov 02 15:11:27.671414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.671344    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:27.721836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.721800    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"]
Nov 02 15:11:27.723590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.723565    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:27.766200 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.766162    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df4cae22-wrl77\" (UID: \"707379e5-d3ea-4f57-89fa-27e0b659290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:27.766408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.766304    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df4cae22-wrl77\" (UID: \"707379e5-d3ea-4f57-89fa-27e0b659290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:27.766472 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.766446    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n49k9\" (UniqueName: \"kubernetes.io/projected/707379e5-d3ea-4f57-89fa-27e0b659290f-kube-api-access-n49k9\") pod \"maas-api-revoke-sub-df4cae22-wrl77\" (UID: \"707379e5-d3ea-4f57-89fa-27e0b659290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:27.767291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.767272    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:11:27.767408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.767383    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:11:27.773086 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.773069    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:11:27.867361 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.867289    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n49k9\" (UniqueName: \"kubernetes.io/projected/707379e5-d3ea-4f57-89fa-27e0b659290f-kube-api-access-n49k9\") pod \"maas-api-revoke-sub-df4cae22-wrl77\" (UID: \"707379e5-d3ea-4f57-89fa-27e0b659290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:27.867567 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.867408    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df4cae22-wrl77\" (UID: \"707379e5-d3ea-4f57-89fa-27e0b659290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:27.867567 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.867504    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df4cae22-wrl77\" (UID: \"707379e5-d3ea-4f57-89fa-27e0b659290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:27.868210 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.868180    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df4cae22-wrl77\" (UID: \"707379e5-d3ea-4f57-89fa-27e0b659290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:27.869810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.869788    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df4cae22-wrl77\" (UID: \"707379e5-d3ea-4f57-89fa-27e0b659290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:27.928493 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.928423    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n49k9\" (UniqueName: \"kubernetes.io/projected/707379e5-d3ea-4f57-89fa-27e0b659290f-kube-api-access-n49k9\") pod \"maas-api-revoke-sub-df4cae22-wrl77\" (UID: \"707379e5-d3ea-4f57-89fa-27e0b659290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:27.988202 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:27.988161    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:28.368989 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:28.368944    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"]
Nov 02 15:11:28.421893 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:11:28.421855    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707379e5_d3ea_4f57_89fa_27e0b659290f.slice/crio-6256857a481773d3c609799d51235a790a1fed88b7ad0da531af24172ee7b6d6 WatchSource:0}: Error finding container 6256857a481773d3c609799d51235a790a1fed88b7ad0da531af24172ee7b6d6: Status 404 returned error can't find the container with id 6256857a481773d3c609799d51235a790a1fed88b7ad0da531af24172ee7b6d6
Nov 02 15:11:28.689616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:28.689583    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77" event={"ID":"707379e5-d3ea-4f57-89fa-27e0b659290f","Type":"ContainerStarted","Data":"6256857a481773d3c609799d51235a790a1fed88b7ad0da531af24172ee7b6d6"}
Nov 02 15:11:29.698431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:29.698348    4891 generic.go:356] "Generic (PLEG): container finished" podID="707379e5-d3ea-4f57-89fa-27e0b659290f" containerID="5e217f303bb1f7643c65352e3aa666a529a1cb8282d180fa575ff547a7f87fc8" exitCode=0
Nov 02 15:11:29.698819 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:29.698441    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77" event={"ID":"707379e5-d3ea-4f57-89fa-27e0b659290f","Type":"ContainerDied","Data":"5e217f303bb1f7643c65352e3aa666a529a1cb8282d180fa575ff547a7f87fc8"}
Nov 02 15:11:30.855826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:30.855799    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:30.903069 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:30.903038    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-openshift-ca\") pod \"707379e5-d3ea-4f57-89fa-27e0b659290f\" (UID: \"707379e5-d3ea-4f57-89fa-27e0b659290f\") "
Nov 02 15:11:30.903262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:30.903082    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-xks-ca\") pod \"707379e5-d3ea-4f57-89fa-27e0b659290f\" (UID: \"707379e5-d3ea-4f57-89fa-27e0b659290f\") "
Nov 02 15:11:30.903465 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:30.903433    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-openshift-ca" pod "707379e5-d3ea-4f57-89fa-27e0b659290f" (UID: "707379e5-d3ea-4f57-89fa-27e0b659290f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:30.905268 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:30.905239    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-xks-ca" pod "707379e5-d3ea-4f57-89fa-27e0b659290f" (UID: "707379e5-d3ea-4f57-89fa-27e0b659290f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:31.004605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.004060    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707379e5-d3ea-4f57-89fa-27e0b659290f-kube-api-access-n49k9\" (UniqueName: \"kubernetes.io/projected/707379e5-d3ea-4f57-89fa-27e0b659290f-kube-api-access-n49k9\") pod \"707379e5-d3ea-4f57-89fa-27e0b659290f\" (UID: \"707379e5-d3ea-4f57-89fa-27e0b659290f\") "
Nov 02 15:11:31.004605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.004507    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:31.004605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.004529    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/707379e5-d3ea-4f57-89fa-27e0b659290f-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:31.006534 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.006501    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707379e5-d3ea-4f57-89fa-27e0b659290f-kube-api-access-n49k9" pod "707379e5-d3ea-4f57-89fa-27e0b659290f" (UID: "707379e5-d3ea-4f57-89fa-27e0b659290f"). InnerVolumeSpecName "kube-api-access-n49k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:31.105429 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.105388    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n49k9\" (UniqueName: \"kubernetes.io/projected/707379e5-d3ea-4f57-89fa-27e0b659290f-kube-api-access-n49k9\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:31.712907 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.712878    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"
Nov 02 15:11:31.713081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.712867    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77" event={"ID":"707379e5-d3ea-4f57-89fa-27e0b659290f","Type":"ContainerDied","Data":"6256857a481773d3c609799d51235a790a1fed88b7ad0da531af24172ee7b6d6"}
Nov 02 15:11:31.713081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.713010    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6256857a481773d3c609799d51235a790a1fed88b7ad0da531af24172ee7b6d6"
Nov 02 15:11:31.910779 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.910743    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"]
Nov 02 15:11:31.911198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.911180    4891 cpu_manager.go:414] "containerMap: removing container" podUID="707379e5-d3ea-4f57-89fa-27e0b659290f" containerName="revoke-keys"
Nov 02 15:11:31.911242 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.911199    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707379e5-d3ea-4f57-89fa-27e0b659290f" containerName="revoke-keys"
Nov 02 15:11:31.911324 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.911298    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="707379e5-d3ea-4f57-89fa-27e0b659290f" containerName="revoke-keys"
Nov 02 15:11:31.914777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.914760    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:31.969033 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.968759    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:31.969033 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.968800    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:11:31.969033 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.968810    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:11:31.969033 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:31.968877    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:11:32.013999 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.013962    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6fdd986-5tddp\" (UID: \"26c9fe1b-828d-4328-a553-ccdf0498673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:32.014176 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.014077    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6fdd986-5tddp\" (UID: \"26c9fe1b-828d-4328-a553-ccdf0498673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:32.014176 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.014144    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2zml\" (UniqueName: \"kubernetes.io/projected/26c9fe1b-828d-4328-a553-ccdf0498673a-kube-api-access-n2zml\") pod \"maas-api-revoke-sub-c6fdd986-5tddp\" (UID: \"26c9fe1b-828d-4328-a553-ccdf0498673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:32.021205 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.021174    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"]
Nov 02 15:11:32.116335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.115449    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6fdd986-5tddp\" (UID: \"26c9fe1b-828d-4328-a553-ccdf0498673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:32.116335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.115548    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6fdd986-5tddp\" (UID: \"26c9fe1b-828d-4328-a553-ccdf0498673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:32.116335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.115582    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2zml\" (UniqueName: \"kubernetes.io/projected/26c9fe1b-828d-4328-a553-ccdf0498673a-kube-api-access-n2zml\") pod \"maas-api-revoke-sub-c6fdd986-5tddp\" (UID: \"26c9fe1b-828d-4328-a553-ccdf0498673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:32.118406 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.117231    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6fdd986-5tddp\" (UID: \"26c9fe1b-828d-4328-a553-ccdf0498673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:32.119791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.119760    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6fdd986-5tddp\" (UID: \"26c9fe1b-828d-4328-a553-ccdf0498673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:32.176256 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.176227    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2zml\" (UniqueName: \"kubernetes.io/projected/26c9fe1b-828d-4328-a553-ccdf0498673a-kube-api-access-n2zml\") pod \"maas-api-revoke-sub-c6fdd986-5tddp\" (UID: \"26c9fe1b-828d-4328-a553-ccdf0498673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:32.228761 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.228677    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:32.708240 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.708207    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"]
Nov 02 15:11:32.740529 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.740495    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp" event={"ID":"26c9fe1b-828d-4328-a553-ccdf0498673a","Type":"ContainerStarted","Data":"26e2891f6fda28c599529a4ffaa7408352710c8b5c46eabf9c83d031283cddc5"}
Nov 02 15:11:32.752666 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.752638    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:11:32.759832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.759806    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:11:32.768433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.760240    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:11:32.775023 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.774997    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:11:32.889334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.885702    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:11:32.912411 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:32.909225    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:11:33.748626 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:33.748590    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp" event={"ID":"26c9fe1b-828d-4328-a553-ccdf0498673a","Type":"ContainerStarted","Data":"2324d3f39c2c56bf0c10723a0da92d312194095c1fc4096df207fff613f5aaa6"}
Nov 02 15:11:33.866158 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:33.866104    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp" podStartSLOduration=2.866089642 podStartE2EDuration="2.866089642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:33.863562278 +0000 UTC m=+901.456179077" watchObservedRunningTime="2026-11-02 15:11:33.866089642 +0000 UTC m=+901.458706438"
Nov 02 15:11:34.756167 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:34.756132    4891 generic.go:356] "Generic (PLEG): container finished" podID="26c9fe1b-828d-4328-a553-ccdf0498673a" containerID="2324d3f39c2c56bf0c10723a0da92d312194095c1fc4096df207fff613f5aaa6" exitCode=0
Nov 02 15:11:34.756612 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:34.756216    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp" event={"ID":"26c9fe1b-828d-4328-a553-ccdf0498673a","Type":"ContainerDied","Data":"2324d3f39c2c56bf0c10723a0da92d312194095c1fc4096df207fff613f5aaa6"}
Nov 02 15:11:35.918458 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:35.918420    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:36.058706 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:36.058672    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-xks-ca\") pod \"26c9fe1b-828d-4328-a553-ccdf0498673a\" (UID: \"26c9fe1b-828d-4328-a553-ccdf0498673a\") "
Nov 02 15:11:36.058916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:36.058806    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c9fe1b-828d-4328-a553-ccdf0498673a-kube-api-access-n2zml\" (UniqueName: \"kubernetes.io/projected/26c9fe1b-828d-4328-a553-ccdf0498673a-kube-api-access-n2zml\") pod \"26c9fe1b-828d-4328-a553-ccdf0498673a\" (UID: \"26c9fe1b-828d-4328-a553-ccdf0498673a\") "
Nov 02 15:11:36.058916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:36.058853    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-openshift-ca\") pod \"26c9fe1b-828d-4328-a553-ccdf0498673a\" (UID: \"26c9fe1b-828d-4328-a553-ccdf0498673a\") "
Nov 02 15:11:36.059267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:36.059244    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-openshift-ca" pod "26c9fe1b-828d-4328-a553-ccdf0498673a" (UID: "26c9fe1b-828d-4328-a553-ccdf0498673a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:36.060852 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:36.060825    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-xks-ca" pod "26c9fe1b-828d-4328-a553-ccdf0498673a" (UID: "26c9fe1b-828d-4328-a553-ccdf0498673a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:36.060965 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:36.060885    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c9fe1b-828d-4328-a553-ccdf0498673a-kube-api-access-n2zml" pod "26c9fe1b-828d-4328-a553-ccdf0498673a" (UID: "26c9fe1b-828d-4328-a553-ccdf0498673a"). InnerVolumeSpecName "kube-api-access-n2zml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:36.168000 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:36.167958    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2zml\" (UniqueName: \"kubernetes.io/projected/26c9fe1b-828d-4328-a553-ccdf0498673a-kube-api-access-n2zml\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:36.168000 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:36.167996    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:36.168000 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:36.168008    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c9fe1b-828d-4328-a553-ccdf0498673a-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:36.772681 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:36.772649    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"
Nov 02 15:11:36.772681 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:36.772662    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp" event={"ID":"26c9fe1b-828d-4328-a553-ccdf0498673a","Type":"ContainerDied","Data":"26e2891f6fda28c599529a4ffaa7408352710c8b5c46eabf9c83d031283cddc5"}
Nov 02 15:11:36.772884 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:36.772715    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26e2891f6fda28c599529a4ffaa7408352710c8b5c46eabf9c83d031283cddc5"
Nov 02 15:11:43.052730 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.052686    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"]
Nov 02 15:11:43.053348 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.053328    4891 cpu_manager.go:414] "containerMap: removing container" podUID="26c9fe1b-828d-4328-a553-ccdf0498673a" containerName="revoke-keys"
Nov 02 15:11:43.053433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.053350    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c9fe1b-828d-4328-a553-ccdf0498673a" containerName="revoke-keys"
Nov 02 15:11:43.053509 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.053493    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c9fe1b-828d-4328-a553-ccdf0498673a" containerName="revoke-keys"
Nov 02 15:11:43.057056 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.057035    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:43.110348 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.110299    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:11:43.114064 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.114031    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"]
Nov 02 15:11:43.124909 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.124879    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf5104d4-tnmq9\" (UID: \"51f75a66-4ca5-4463-836f-d1120ae748db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:43.125140 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.125036    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4hdt\" (UniqueName: \"kubernetes.io/projected/51f75a66-4ca5-4463-836f-d1120ae748db-kube-api-access-q4hdt\") pod \"maas-api-revoke-sub-bf5104d4-tnmq9\" (UID: \"51f75a66-4ca5-4463-836f-d1120ae748db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:43.125236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.125154    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf5104d4-tnmq9\" (UID: \"51f75a66-4ca5-4463-836f-d1120ae748db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:43.156844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.156812    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:11:43.159672 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.159640    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:43.159788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.159701    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:11:43.226013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.225977    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf5104d4-tnmq9\" (UID: \"51f75a66-4ca5-4463-836f-d1120ae748db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:43.226224 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.226073    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4hdt\" (UniqueName: \"kubernetes.io/projected/51f75a66-4ca5-4463-836f-d1120ae748db-kube-api-access-q4hdt\") pod \"maas-api-revoke-sub-bf5104d4-tnmq9\" (UID: \"51f75a66-4ca5-4463-836f-d1120ae748db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:43.226224 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.226112    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf5104d4-tnmq9\" (UID: \"51f75a66-4ca5-4463-836f-d1120ae748db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:43.226949 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.226927    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf5104d4-tnmq9\" (UID: \"51f75a66-4ca5-4463-836f-d1120ae748db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:43.228580 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.228558    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf5104d4-tnmq9\" (UID: \"51f75a66-4ca5-4463-836f-d1120ae748db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:43.287439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.287407    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4hdt\" (UniqueName: \"kubernetes.io/projected/51f75a66-4ca5-4463-836f-d1120ae748db-kube-api-access-q4hdt\") pod \"maas-api-revoke-sub-bf5104d4-tnmq9\" (UID: \"51f75a66-4ca5-4463-836f-d1120ae748db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:43.370913 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.370833    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:43.760469 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.760438    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"]
Nov 02 15:11:43.762459 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:11:43.762430    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f75a66_4ca5_4463_836f_d1120ae748db.slice/crio-b6046acc9e34c430b481d0423cfe345cc9c94c9ee1d3992a5d320984d1b8ec94 WatchSource:0}: Error finding container b6046acc9e34c430b481d0423cfe345cc9c94c9ee1d3992a5d320984d1b8ec94: Status 404 returned error can't find the container with id b6046acc9e34c430b481d0423cfe345cc9c94c9ee1d3992a5d320984d1b8ec94
Nov 02 15:11:43.819427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:43.819393    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9" event={"ID":"51f75a66-4ca5-4463-836f-d1120ae748db","Type":"ContainerStarted","Data":"b6046acc9e34c430b481d0423cfe345cc9c94c9ee1d3992a5d320984d1b8ec94"}
Nov 02 15:11:44.274153 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:11:44.274119    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:11:44.826717 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:44.826673    4891 generic.go:356] "Generic (PLEG): container finished" podID="51f75a66-4ca5-4463-836f-d1120ae748db" containerID="1bd32e374cfe9d80138cf38887f0b4584af26291cbeee8ab04d4e457d6ca7bdf" exitCode=0
Nov 02 15:11:44.826918 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:44.826824    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9" event={"ID":"51f75a66-4ca5-4463-836f-d1120ae748db","Type":"ContainerDied","Data":"1bd32e374cfe9d80138cf38887f0b4584af26291cbeee8ab04d4e457d6ca7bdf"}
Nov 02 15:11:45.986024 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:45.985993    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:46.059691 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:46.059647    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-openshift-ca\") pod \"51f75a66-4ca5-4463-836f-d1120ae748db\" (UID: \"51f75a66-4ca5-4463-836f-d1120ae748db\") "
Nov 02 15:11:46.059890 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:46.059751    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f75a66-4ca5-4463-836f-d1120ae748db-kube-api-access-q4hdt\" (UniqueName: \"kubernetes.io/projected/51f75a66-4ca5-4463-836f-d1120ae748db-kube-api-access-q4hdt\") pod \"51f75a66-4ca5-4463-836f-d1120ae748db\" (UID: \"51f75a66-4ca5-4463-836f-d1120ae748db\") "
Nov 02 15:11:46.059890 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:46.059782    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-xks-ca\") pod \"51f75a66-4ca5-4463-836f-d1120ae748db\" (UID: \"51f75a66-4ca5-4463-836f-d1120ae748db\") "
Nov 02 15:11:46.060122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:46.060095    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-openshift-ca" pod "51f75a66-4ca5-4463-836f-d1120ae748db" (UID: "51f75a66-4ca5-4463-836f-d1120ae748db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:46.061821 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:46.061795    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f75a66-4ca5-4463-836f-d1120ae748db-kube-api-access-q4hdt" pod "51f75a66-4ca5-4463-836f-d1120ae748db" (UID: "51f75a66-4ca5-4463-836f-d1120ae748db"). InnerVolumeSpecName "kube-api-access-q4hdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:46.061900 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:46.061884    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-xks-ca" pod "51f75a66-4ca5-4463-836f-d1120ae748db" (UID: "51f75a66-4ca5-4463-836f-d1120ae748db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:46.161263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:46.161178    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:46.161263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:46.161209    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4hdt\" (UniqueName: \"kubernetes.io/projected/51f75a66-4ca5-4463-836f-d1120ae748db-kube-api-access-q4hdt\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:46.161263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:46.161220    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f75a66-4ca5-4463-836f-d1120ae748db-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:46.841423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:46.841339    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"
Nov 02 15:11:46.841423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:46.841355    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9" event={"ID":"51f75a66-4ca5-4463-836f-d1120ae748db","Type":"ContainerDied","Data":"b6046acc9e34c430b481d0423cfe345cc9c94c9ee1d3992a5d320984d1b8ec94"}
Nov 02 15:11:46.841423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:46.841408    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6046acc9e34c430b481d0423cfe345cc9c94c9ee1d3992a5d320984d1b8ec94"
Nov 02 15:11:52.881291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.881254    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"]
Nov 02 15:11:52.881833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.881816    4891 cpu_manager.go:414] "containerMap: removing container" podUID="51f75a66-4ca5-4463-836f-d1120ae748db" containerName="revoke-keys"
Nov 02 15:11:52.881908 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.881835    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f75a66-4ca5-4463-836f-d1120ae748db" containerName="revoke-keys"
Nov 02 15:11:52.881975 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.881965    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f75a66-4ca5-4463-836f-d1120ae748db" containerName="revoke-keys"
Nov 02 15:11:52.885535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.885503    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:52.934355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.934298    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p99hm\" (UniqueName: \"kubernetes.io/projected/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-kube-api-access-p99hm\") pod \"maas-api-revoke-sub-43407818-rrjkn\" (UID: \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:52.934526 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.934380    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43407818-rrjkn\" (UID: \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:52.934526 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.934421    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43407818-rrjkn\" (UID: \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:52.942658 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.942629    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"]
Nov 02 15:11:52.988097 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.988064    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:11:52.988097 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.988065    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:11:52.988097 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.988064    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:52.988388 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:52.988165    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:11:53.035754 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:53.035721    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p99hm\" (UniqueName: \"kubernetes.io/projected/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-kube-api-access-p99hm\") pod \"maas-api-revoke-sub-43407818-rrjkn\" (UID: \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:53.035902 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:53.035770    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43407818-rrjkn\" (UID: \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:53.035902 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:53.035837    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43407818-rrjkn\" (UID: \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:53.036442 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:53.036420    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43407818-rrjkn\" (UID: \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:53.038212 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:53.038192    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43407818-rrjkn\" (UID: \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:53.097881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:53.097843    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p99hm\" (UniqueName: \"kubernetes.io/projected/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-kube-api-access-p99hm\") pod \"maas-api-revoke-sub-43407818-rrjkn\" (UID: \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:53.201214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:53.201129    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:53.578981 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:53.578947    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"]
Nov 02 15:11:53.890477 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:53.890435    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn" event={"ID":"975e03dd-1501-4a43-ac9b-0c9beba3cf0d","Type":"ContainerStarted","Data":"776fac556f36908e24e4520157a045ea56d59304d292dcc1f987fd19993f8c06"}
Nov 02 15:11:54.898274 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:54.898235    4891 generic.go:356] "Generic (PLEG): container finished" podID="975e03dd-1501-4a43-ac9b-0c9beba3cf0d" containerID="91d55f249045a69654dfa62dd1035dd0b6b65807b432922c3c27a7eb397f4828" exitCode=0
Nov 02 15:11:54.898732 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:54.898329    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn" event={"ID":"975e03dd-1501-4a43-ac9b-0c9beba3cf0d","Type":"ContainerDied","Data":"91d55f249045a69654dfa62dd1035dd0b6b65807b432922c3c27a7eb397f4828"}
Nov 02 15:11:56.056317 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.056282    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:56.171466 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.171391    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-xks-ca\") pod \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\" (UID: \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\") "
Nov 02 15:11:56.171659 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.171509    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-kube-api-access-p99hm\" (UniqueName: \"kubernetes.io/projected/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-kube-api-access-p99hm\") pod \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\" (UID: \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\") "
Nov 02 15:11:56.171659 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.171630    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-openshift-ca\") pod \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\" (UID: \"975e03dd-1501-4a43-ac9b-0c9beba3cf0d\") "
Nov 02 15:11:56.171991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.171954    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-openshift-ca" pod "975e03dd-1501-4a43-ac9b-0c9beba3cf0d" (UID: "975e03dd-1501-4a43-ac9b-0c9beba3cf0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:56.172109 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.172050    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:56.174105 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.174077    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-xks-ca" pod "975e03dd-1501-4a43-ac9b-0c9beba3cf0d" (UID: "975e03dd-1501-4a43-ac9b-0c9beba3cf0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:56.174213 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.174109    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-kube-api-access-p99hm" pod "975e03dd-1501-4a43-ac9b-0c9beba3cf0d" (UID: "975e03dd-1501-4a43-ac9b-0c9beba3cf0d"). InnerVolumeSpecName "kube-api-access-p99hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:56.273104 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.273068    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:56.273104 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.273101    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p99hm\" (UniqueName: \"kubernetes.io/projected/975e03dd-1501-4a43-ac9b-0c9beba3cf0d-kube-api-access-p99hm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:11:56.914295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.914208    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"
Nov 02 15:11:56.914295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.914230    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn" event={"ID":"975e03dd-1501-4a43-ac9b-0c9beba3cf0d","Type":"ContainerDied","Data":"776fac556f36908e24e4520157a045ea56d59304d292dcc1f987fd19993f8c06"}
Nov 02 15:11:56.914495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:56.914298    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="776fac556f36908e24e4520157a045ea56d59304d292dcc1f987fd19993f8c06"
Nov 02 15:11:57.477592 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.477553    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"]
Nov 02 15:11:57.478090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.478033    4891 cpu_manager.go:414] "containerMap: removing container" podUID="975e03dd-1501-4a43-ac9b-0c9beba3cf0d" containerName="revoke-keys"
Nov 02 15:11:57.478090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.478045    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975e03dd-1501-4a43-ac9b-0c9beba3cf0d" containerName="revoke-keys"
Nov 02 15:11:57.478205 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.478120    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="975e03dd-1501-4a43-ac9b-0c9beba3cf0d" containerName="revoke-keys"
Nov 02 15:11:57.481522 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.481498    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:11:57.585125 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.585088    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:11:57.585352 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.585089    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:11:57.585470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.585161    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:57.585619 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.585584    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:11:57.588151 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.588114    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzgdt\" (UniqueName: \"kubernetes.io/projected/6413a9f1-1ab5-4970-a969-f343db4dbd7d-kube-api-access-rzgdt\") pod \"maas-api-revoke-sub-fdc04803-b6k6z\" (UID: \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:11:57.588327 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.588167    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc04803-b6k6z\" (UID: \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:11:57.588327 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.588285    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc04803-b6k6z\" (UID: \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:11:57.589448 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.589422    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"]
Nov 02 15:11:57.689815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.689778    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc04803-b6k6z\" (UID: \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:11:57.690025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.689906    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzgdt\" (UniqueName: \"kubernetes.io/projected/6413a9f1-1ab5-4970-a969-f343db4dbd7d-kube-api-access-rzgdt\") pod \"maas-api-revoke-sub-fdc04803-b6k6z\" (UID: \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:11:57.690025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.689939    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc04803-b6k6z\" (UID: \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:11:57.690799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.690758    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc04803-b6k6z\" (UID: \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:11:57.692443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.692425    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc04803-b6k6z\" (UID: \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:11:57.755271 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.755181    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzgdt\" (UniqueName: \"kubernetes.io/projected/6413a9f1-1ab5-4970-a969-f343db4dbd7d-kube-api-access-rzgdt\") pod \"maas-api-revoke-sub-fdc04803-b6k6z\" (UID: \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:11:57.799527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:57.799488    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:11:58.272084 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:58.272057    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"]
Nov 02 15:11:58.274352 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:11:58.274289    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6413a9f1_1ab5_4970_a969_f343db4dbd7d.slice/crio-b1768dad92b55fd8878f9334b6a550491cb0d89404f339e961661aaa0e4d3849 WatchSource:0}: Error finding container b1768dad92b55fd8878f9334b6a550491cb0d89404f339e961661aaa0e4d3849: Status 404 returned error can't find the container with id b1768dad92b55fd8878f9334b6a550491cb0d89404f339e961661aaa0e4d3849
Nov 02 15:11:58.971375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:58.955975    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"]
Nov 02 15:11:58.971375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:58.960557    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:11:59.014072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.014037    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d14500a-gpggr\" (UID: \"30570f1a-79a9-4468-acb0-1dccc3530e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:11:59.015481 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.015435    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d14500a-gpggr\" (UID: \"30570f1a-79a9-4468-acb0-1dccc3530e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:11:59.015822 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.015799    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97lzg\" (UniqueName: \"kubernetes.io/projected/30570f1a-79a9-4468-acb0-1dccc3530e5e-kube-api-access-97lzg\") pod \"maas-api-revoke-sub-5d14500a-gpggr\" (UID: \"30570f1a-79a9-4468-acb0-1dccc3530e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:11:59.017750 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.017721    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z" event={"ID":"6413a9f1-1ab5-4970-a969-f343db4dbd7d","Type":"ContainerStarted","Data":"74378803719b4c251f23157cb4260b28c3593894fe155a711115ebbdbaa54414"}
Nov 02 15:11:59.017853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.017781    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z" event={"ID":"6413a9f1-1ab5-4970-a969-f343db4dbd7d","Type":"ContainerStarted","Data":"b1768dad92b55fd8878f9334b6a550491cb0d89404f339e961661aaa0e4d3849"}
Nov 02 15:11:59.021199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.021112    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"]
Nov 02 15:11:59.117714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.117680    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97lzg\" (UniqueName: \"kubernetes.io/projected/30570f1a-79a9-4468-acb0-1dccc3530e5e-kube-api-access-97lzg\") pod \"maas-api-revoke-sub-5d14500a-gpggr\" (UID: \"30570f1a-79a9-4468-acb0-1dccc3530e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:11:59.117988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.117974    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d14500a-gpggr\" (UID: \"30570f1a-79a9-4468-acb0-1dccc3530e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:11:59.118177 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.118166    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d14500a-gpggr\" (UID: \"30570f1a-79a9-4468-acb0-1dccc3530e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:11:59.118962 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.118927    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d14500a-gpggr\" (UID: \"30570f1a-79a9-4468-acb0-1dccc3530e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:11:59.120805 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.120785    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d14500a-gpggr\" (UID: \"30570f1a-79a9-4468-acb0-1dccc3530e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:11:59.203727 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.203691    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97lzg\" (UniqueName: \"kubernetes.io/projected/30570f1a-79a9-4468-acb0-1dccc3530e5e-kube-api-access-97lzg\") pod \"maas-api-revoke-sub-5d14500a-gpggr\" (UID: \"30570f1a-79a9-4468-acb0-1dccc3530e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:11:59.214169 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.214118    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z" podStartSLOduration=2.214102831 podStartE2EDuration="2.214102831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:59.211879308 +0000 UTC m=+926.804496110" watchObservedRunningTime="2026-11-02 15:11:59.214102831 +0000 UTC m=+926.806719667"
Nov 02 15:11:59.318837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.318800    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:11:59.751913 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:11:59.751875    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"]
Nov 02 15:12:00.025840 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:00.025806    4891 generic.go:356] "Generic (PLEG): container finished" podID="6413a9f1-1ab5-4970-a969-f343db4dbd7d" containerID="74378803719b4c251f23157cb4260b28c3593894fe155a711115ebbdbaa54414" exitCode=0
Nov 02 15:12:00.026248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:00.025898    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z" event={"ID":"6413a9f1-1ab5-4970-a969-f343db4dbd7d","Type":"ContainerDied","Data":"74378803719b4c251f23157cb4260b28c3593894fe155a711115ebbdbaa54414"}
Nov 02 15:12:00.027549 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:00.027527    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr" event={"ID":"30570f1a-79a9-4468-acb0-1dccc3530e5e","Type":"ContainerStarted","Data":"745ea6eac9a518355c85e1e272e39ce8c7e83ec2fdebb7942ebbdcc5ddfba335"}
Nov 02 15:12:01.035634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:01.035597    4891 generic.go:356] "Generic (PLEG): container finished" podID="30570f1a-79a9-4468-acb0-1dccc3530e5e" containerID="6bfdef5f85e2b598549e50b96c6ec35b357fbde3992719431dd18d25ca06179a" exitCode=0
Nov 02 15:12:01.036031 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:01.035695    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr" event={"ID":"30570f1a-79a9-4468-acb0-1dccc3530e5e","Type":"ContainerDied","Data":"6bfdef5f85e2b598549e50b96c6ec35b357fbde3992719431dd18d25ca06179a"}
Nov 02 15:12:01.195700 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:01.195664    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:12:01.239519 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:01.239479    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-openshift-ca\") pod \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\" (UID: \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\") "
Nov 02 15:12:01.239665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:01.239583    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6413a9f1-1ab5-4970-a969-f343db4dbd7d-kube-api-access-rzgdt\" (UniqueName: \"kubernetes.io/projected/6413a9f1-1ab5-4970-a969-f343db4dbd7d-kube-api-access-rzgdt\") pod \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\" (UID: \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\") "
Nov 02 15:12:01.239760 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:01.239743    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-xks-ca\") pod \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\" (UID: \"6413a9f1-1ab5-4970-a969-f343db4dbd7d\") "
Nov 02 15:12:01.239849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:01.239822    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-openshift-ca" pod "6413a9f1-1ab5-4970-a969-f343db4dbd7d" (UID: "6413a9f1-1ab5-4970-a969-f343db4dbd7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:01.240137 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:01.240112    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:01.241731 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:01.241706    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6413a9f1-1ab5-4970-a969-f343db4dbd7d-kube-api-access-rzgdt" pod "6413a9f1-1ab5-4970-a969-f343db4dbd7d" (UID: "6413a9f1-1ab5-4970-a969-f343db4dbd7d"). InnerVolumeSpecName "kube-api-access-rzgdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:01.241824 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:01.241710    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-xks-ca" pod "6413a9f1-1ab5-4970-a969-f343db4dbd7d" (UID: "6413a9f1-1ab5-4970-a969-f343db4dbd7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:01.341989 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:01.341896    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzgdt\" (UniqueName: \"kubernetes.io/projected/6413a9f1-1ab5-4970-a969-f343db4dbd7d-kube-api-access-rzgdt\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:01.341989 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:01.341944    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6413a9f1-1ab5-4970-a969-f343db4dbd7d-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:02.043435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.043400    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"
Nov 02 15:12:02.043435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.043404    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z" event={"ID":"6413a9f1-1ab5-4970-a969-f343db4dbd7d","Type":"ContainerDied","Data":"b1768dad92b55fd8878f9334b6a550491cb0d89404f339e961661aaa0e4d3849"}
Nov 02 15:12:02.043912 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.043479    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1768dad92b55fd8878f9334b6a550491cb0d89404f339e961661aaa0e4d3849"
Nov 02 15:12:02.187495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.187465    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:12:02.251679 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.251641    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30570f1a-79a9-4468-acb0-1dccc3530e5e-kube-api-access-97lzg\" (UniqueName: \"kubernetes.io/projected/30570f1a-79a9-4468-acb0-1dccc3530e5e-kube-api-access-97lzg\") pod \"30570f1a-79a9-4468-acb0-1dccc3530e5e\" (UID: \"30570f1a-79a9-4468-acb0-1dccc3530e5e\") "
Nov 02 15:12:02.251868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.251705    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-openshift-ca\") pod \"30570f1a-79a9-4468-acb0-1dccc3530e5e\" (UID: \"30570f1a-79a9-4468-acb0-1dccc3530e5e\") "
Nov 02 15:12:02.251868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.251780    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-xks-ca\") pod \"30570f1a-79a9-4468-acb0-1dccc3530e5e\" (UID: \"30570f1a-79a9-4468-acb0-1dccc3530e5e\") "
Nov 02 15:12:02.252109 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.252085    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-openshift-ca" pod "30570f1a-79a9-4468-acb0-1dccc3530e5e" (UID: "30570f1a-79a9-4468-acb0-1dccc3530e5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:02.253833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.253805    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30570f1a-79a9-4468-acb0-1dccc3530e5e-kube-api-access-97lzg" pod "30570f1a-79a9-4468-acb0-1dccc3530e5e" (UID: "30570f1a-79a9-4468-acb0-1dccc3530e5e"). InnerVolumeSpecName "kube-api-access-97lzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:02.253939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.253902    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-xks-ca" pod "30570f1a-79a9-4468-acb0-1dccc3530e5e" (UID: "30570f1a-79a9-4468-acb0-1dccc3530e5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:02.353567 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.353478    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:02.353567 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.353514    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97lzg\" (UniqueName: \"kubernetes.io/projected/30570f1a-79a9-4468-acb0-1dccc3530e5e-kube-api-access-97lzg\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:02.353567 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.353528    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30570f1a-79a9-4468-acb0-1dccc3530e5e-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:02.953796 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.953756    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"]
Nov 02 15:12:02.954214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.954201    4891 cpu_manager.go:414] "containerMap: removing container" podUID="30570f1a-79a9-4468-acb0-1dccc3530e5e" containerName="revoke-keys"
Nov 02 15:12:02.954277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.954215    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30570f1a-79a9-4468-acb0-1dccc3530e5e" containerName="revoke-keys"
Nov 02 15:12:02.954277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.954229    4891 cpu_manager.go:414] "containerMap: removing container" podUID="6413a9f1-1ab5-4970-a969-f343db4dbd7d" containerName="revoke-keys"
Nov 02 15:12:02.954277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.954234    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6413a9f1-1ab5-4970-a969-f343db4dbd7d" containerName="revoke-keys"
Nov 02 15:12:02.954444 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.954342    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="30570f1a-79a9-4468-acb0-1dccc3530e5e" containerName="revoke-keys"
Nov 02 15:12:02.954444 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.954352    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="6413a9f1-1ab5-4970-a969-f343db4dbd7d" containerName="revoke-keys"
Nov 02 15:12:02.957638 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:02.957621    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.010995 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.010959    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-db3931-openshift-default-dockercfg-w9t9c\""
Nov 02 15:12:03.050800 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.050770    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"
Nov 02 15:12:03.051267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.050794    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr" event={"ID":"30570f1a-79a9-4468-acb0-1dccc3530e5e","Type":"ContainerDied","Data":"745ea6eac9a518355c85e1e272e39ce8c7e83ec2fdebb7942ebbdcc5ddfba335"}
Nov 02 15:12:03.051406 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.051292    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="745ea6eac9a518355c85e1e272e39ce8c7e83ec2fdebb7942ebbdcc5ddfba335"
Nov 02 15:12:03.061291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.061253    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.061472 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.061421    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.061472 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.061461    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.061649 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.061629    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.061712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.061673    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7304fe30-39ac-4e9a-b880-20cc50287018-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.061795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.061772    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.061872 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.061858    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.061926 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.061893    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55f4s\" (UniqueName: \"kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-kube-api-access-55f4s\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.061979 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.061959    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7304fe30-39ac-4e9a-b880-20cc50287018-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.069175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.069141    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"]
Nov 02 15:12:03.163095 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163063    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163103    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7304fe30-39ac-4e9a-b880-20cc50287018-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163152    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163190    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163218    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55f4s\" (UniqueName: \"kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-kube-api-access-55f4s\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163260    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7304fe30-39ac-4e9a-b880-20cc50287018-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.163587 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163300    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.163587 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163404    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.163587 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163442    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.163587 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163547    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.163770 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163680    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.163853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163830    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.163895 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.163866    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.164096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.164078    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7304fe30-39ac-4e9a-b880-20cc50287018-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.165652 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.165633    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.165778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.165759    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7304fe30-39ac-4e9a-b880-20cc50287018-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.224699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.224625    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.224916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.224889    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55f4s\" (UniqueName: \"kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-kube-api-access-55f4s\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.271244 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.271206    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:03.720815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.720773    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"]
Nov 02 15:12:03.721719 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:03.721692    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7304fe30_39ac_4e9a_b880_20cc50287018.slice/crio-4b40132c748387d5d62b67ea89e837dab49514d17eb0e86054aa8e5ddfb0fe82 WatchSource:0}: Error finding container 4b40132c748387d5d62b67ea89e837dab49514d17eb0e86054aa8e5ddfb0fe82: Status 404 returned error can't find the container with id 4b40132c748387d5d62b67ea89e837dab49514d17eb0e86054aa8e5ddfb0fe82
Nov 02 15:12:03.724093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.724061    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:03.724180 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.724121    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:03.724180 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.724149    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:03.797567 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.797537    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"]
Nov 02 15:12:03.801269 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.801244    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:03.861897 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.861863    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:12:03.862082 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.861976    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:12:03.862082 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.862014    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:12:03.862189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.862139    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:12:03.866810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.866775    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"]
Nov 02 15:12:03.878675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.878637    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dd244b3-bsz9q\" (UID: \"a3e41390-f485-4aa2-8fa6-44e029042712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:03.878863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.878768    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v47j\" (UniqueName: \"kubernetes.io/projected/a3e41390-f485-4aa2-8fa6-44e029042712-kube-api-access-8v47j\") pod \"maas-api-revoke-sub-9dd244b3-bsz9q\" (UID: \"a3e41390-f485-4aa2-8fa6-44e029042712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:03.878863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.878851    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dd244b3-bsz9q\" (UID: \"a3e41390-f485-4aa2-8fa6-44e029042712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:03.979703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.979599    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dd244b3-bsz9q\" (UID: \"a3e41390-f485-4aa2-8fa6-44e029042712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:03.979862 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.979724    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dd244b3-bsz9q\" (UID: \"a3e41390-f485-4aa2-8fa6-44e029042712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:03.979862 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.979810    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v47j\" (UniqueName: \"kubernetes.io/projected/a3e41390-f485-4aa2-8fa6-44e029042712-kube-api-access-8v47j\") pod \"maas-api-revoke-sub-9dd244b3-bsz9q\" (UID: \"a3e41390-f485-4aa2-8fa6-44e029042712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:03.980507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.980479    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dd244b3-bsz9q\" (UID: \"a3e41390-f485-4aa2-8fa6-44e029042712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:03.981990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:03.981961    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dd244b3-bsz9q\" (UID: \"a3e41390-f485-4aa2-8fa6-44e029042712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:04.041127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:04.041094    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v47j\" (UniqueName: \"kubernetes.io/projected/a3e41390-f485-4aa2-8fa6-44e029042712-kube-api-access-8v47j\") pod \"maas-api-revoke-sub-9dd244b3-bsz9q\" (UID: \"a3e41390-f485-4aa2-8fa6-44e029042712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:04.059109 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:04.059064    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" event={"ID":"7304fe30-39ac-4e9a-b880-20cc50287018","Type":"ContainerStarted","Data":"72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5"}
Nov 02 15:12:04.059559 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:04.059125    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" event={"ID":"7304fe30-39ac-4e9a-b880-20cc50287018","Type":"ContainerStarted","Data":"4b40132c748387d5d62b67ea89e837dab49514d17eb0e86054aa8e5ddfb0fe82"}
Nov 02 15:12:04.118535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:04.118498    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:04.185106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:04.182946    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" podStartSLOduration=2.182932605 podStartE2EDuration="2.182932605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:04.179673379 +0000 UTC m=+931.772290165" watchObservedRunningTime="2026-11-02 15:12:04.182932605 +0000 UTC m=+931.775549403"
Nov 02 15:12:04.271936 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:04.271907    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:04.273284 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:04.273261    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:04.273430 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:04.273306    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" podUID="7304fe30-39ac-4e9a-b880-20cc50287018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 15:12:04.498815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:04.498785    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"]
Nov 02 15:12:04.500109 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:04.500078    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e41390_f485_4aa2_8fa6_44e029042712.slice/crio-552df0f851662fdc169f3ce902f71bc5be07904736447d4087858b07bf0f8b79 WatchSource:0}: Error finding container 552df0f851662fdc169f3ce902f71bc5be07904736447d4087858b07bf0f8b79: Status 404 returned error can't find the container with id 552df0f851662fdc169f3ce902f71bc5be07904736447d4087858b07bf0f8b79
Nov 02 15:12:05.077233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:05.077196    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q" event={"ID":"a3e41390-f485-4aa2-8fa6-44e029042712","Type":"ContainerStarted","Data":"8d0f7d576cd6cb72f0a43c1bd32eb266bd11cbc3d3f1c7aea68855bed1ce0b26"}
Nov 02 15:12:05.077710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:05.077257    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q" event={"ID":"a3e41390-f485-4aa2-8fa6-44e029042712","Type":"ContainerStarted","Data":"552df0f851662fdc169f3ce902f71bc5be07904736447d4087858b07bf0f8b79"}
Nov 02 15:12:05.194360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:05.194286    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q" podStartSLOduration=2.194270243 podStartE2EDuration="2.194270243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:05.192298537 +0000 UTC m=+932.784915335" watchObservedRunningTime="2026-11-02 15:12:05.194270243 +0000 UTC m=+932.786887041"
Nov 02 15:12:05.275876 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:05.275423    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:05.275876 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:05.275489    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" podUID="7304fe30-39ac-4e9a-b880-20cc50287018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 15:12:06.085650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:06.085615    4891 generic.go:356] "Generic (PLEG): container finished" podID="a3e41390-f485-4aa2-8fa6-44e029042712" containerID="8d0f7d576cd6cb72f0a43c1bd32eb266bd11cbc3d3f1c7aea68855bed1ce0b26" exitCode=0
Nov 02 15:12:06.086050 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:06.085691    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q" event={"ID":"a3e41390-f485-4aa2-8fa6-44e029042712","Type":"ContainerDied","Data":"8d0f7d576cd6cb72f0a43c1bd32eb266bd11cbc3d3f1c7aea68855bed1ce0b26"}
Nov 02 15:12:06.271969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:06.271925    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:06.272165 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:06.271994    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" podUID="7304fe30-39ac-4e9a-b880-20cc50287018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 15:12:07.289375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:07.275884    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:07.289375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:07.275965    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" podUID="7304fe30-39ac-4e9a-b880-20cc50287018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 15:12:07.310981 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:07.310953    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:07.420579 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:07.420501    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e41390-f485-4aa2-8fa6-44e029042712-kube-api-access-8v47j\" (UniqueName: \"kubernetes.io/projected/a3e41390-f485-4aa2-8fa6-44e029042712-kube-api-access-8v47j\") pod \"a3e41390-f485-4aa2-8fa6-44e029042712\" (UID: \"a3e41390-f485-4aa2-8fa6-44e029042712\") "
Nov 02 15:12:07.420579 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:07.420549    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-xks-ca\") pod \"a3e41390-f485-4aa2-8fa6-44e029042712\" (UID: \"a3e41390-f485-4aa2-8fa6-44e029042712\") "
Nov 02 15:12:07.420782 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:07.420624    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-openshift-ca\") pod \"a3e41390-f485-4aa2-8fa6-44e029042712\" (UID: \"a3e41390-f485-4aa2-8fa6-44e029042712\") "
Nov 02 15:12:07.420976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:07.420953    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-openshift-ca" pod "a3e41390-f485-4aa2-8fa6-44e029042712" (UID: "a3e41390-f485-4aa2-8fa6-44e029042712"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:07.422640 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:07.422616    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-xks-ca" pod "a3e41390-f485-4aa2-8fa6-44e029042712" (UID: "a3e41390-f485-4aa2-8fa6-44e029042712"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:07.422750 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:07.422648    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e41390-f485-4aa2-8fa6-44e029042712-kube-api-access-8v47j" pod "a3e41390-f485-4aa2-8fa6-44e029042712" (UID: "a3e41390-f485-4aa2-8fa6-44e029042712"). InnerVolumeSpecName "kube-api-access-8v47j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:07.521383 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:07.521350    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v47j\" (UniqueName: \"kubernetes.io/projected/a3e41390-f485-4aa2-8fa6-44e029042712-kube-api-access-8v47j\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:07.521383 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:07.521380    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:07.521383 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:07.521390    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e41390-f485-4aa2-8fa6-44e029042712-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:08.104491 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.104452    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q" event={"ID":"a3e41390-f485-4aa2-8fa6-44e029042712","Type":"ContainerDied","Data":"552df0f851662fdc169f3ce902f71bc5be07904736447d4087858b07bf0f8b79"}
Nov 02 15:12:08.104676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.104508    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="552df0f851662fdc169f3ce902f71bc5be07904736447d4087858b07bf0f8b79"
Nov 02 15:12:08.104676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.104466    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"
Nov 02 15:12:08.232352 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.232317    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"]
Nov 02 15:12:08.232848 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.232831    4891 cpu_manager.go:414] "containerMap: removing container" podUID="a3e41390-f485-4aa2-8fa6-44e029042712" containerName="revoke-keys"
Nov 02 15:12:08.232898 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.232849    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e41390-f485-4aa2-8fa6-44e029042712" containerName="revoke-keys"
Nov 02 15:12:08.232968 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.232958    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e41390-f485-4aa2-8fa6-44e029042712" containerName="revoke-keys"
Nov 02 15:12:08.236551 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.236531    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:08.272991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.272955    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:08.273192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.273017    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" podUID="7304fe30-39ac-4e9a-b880-20cc50287018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 15:12:08.288917 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.288889    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:12:08.289112 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.288978    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:12:08.289112 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.288996    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:12:08.290384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.290365    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:12:08.303197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.303164    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"]
Nov 02 15:12:08.328509 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.328476    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5hs\" (UniqueName: \"kubernetes.io/projected/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-kube-api-access-8m5hs\") pod \"maas-api-revoke-sub-18e0eb47-djzrz\" (UID: \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:08.328684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.328520    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e0eb47-djzrz\" (UID: \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:08.328684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.328588    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e0eb47-djzrz\" (UID: \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:08.429736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.429655    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e0eb47-djzrz\" (UID: \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:08.429736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.429704    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e0eb47-djzrz\" (UID: \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:08.429925 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.429867    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5hs\" (UniqueName: \"kubernetes.io/projected/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-kube-api-access-8m5hs\") pod \"maas-api-revoke-sub-18e0eb47-djzrz\" (UID: \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:08.430391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.430366    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e0eb47-djzrz\" (UID: \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:08.432091 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.432072    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e0eb47-djzrz\" (UID: \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:08.490942 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.490907    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5hs\" (UniqueName: \"kubernetes.io/projected/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-kube-api-access-8m5hs\") pod \"maas-api-revoke-sub-18e0eb47-djzrz\" (UID: \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:08.573484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.573445    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:08.971286 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:08.971254    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"]
Nov 02 15:12:08.973533 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:08.973499    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9a7b60_6e32_4f02_a984_f5e472cb7dea.slice/crio-22b30258920122de71b8d324f202cf31e2290bbf531e09e7792d3ba0b01170ae WatchSource:0}: Error finding container 22b30258920122de71b8d324f202cf31e2290bbf531e09e7792d3ba0b01170ae: Status 404 returned error can't find the container with id 22b30258920122de71b8d324f202cf31e2290bbf531e09e7792d3ba0b01170ae
Nov 02 15:12:09.112284 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:09.112248    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz" event={"ID":"1a9a7b60-6e32-4f02-a984-f5e472cb7dea","Type":"ContainerStarted","Data":"22b30258920122de71b8d324f202cf31e2290bbf531e09e7792d3ba0b01170ae"}
Nov 02 15:12:09.271835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:09.271803    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:09.271977 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:09.271859    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" podUID="7304fe30-39ac-4e9a-b880-20cc50287018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 15:12:10.121445 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:10.121414    4891 generic.go:356] "Generic (PLEG): container finished" podID="1a9a7b60-6e32-4f02-a984-f5e472cb7dea" containerID="1ec33e1af921a898417f7526f0c455ac3089be18edb5637645e1aac117a48475" exitCode=0
Nov 02 15:12:10.121783 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:10.121488    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz" event={"ID":"1a9a7b60-6e32-4f02-a984-f5e472cb7dea","Type":"ContainerDied","Data":"1ec33e1af921a898417f7526f0c455ac3089be18edb5637645e1aac117a48475"}
Nov 02 15:12:10.271811 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:10.271777    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:10.271989 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:10.271828    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" podUID="7304fe30-39ac-4e9a-b880-20cc50287018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 15:12:11.265895 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.265871    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:11.276375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.276353    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:11.276651 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.276628    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:11.277372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.277354    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:12:11.357205 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.357178    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-openshift-ca\") pod \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\" (UID: \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\") "
Nov 02 15:12:11.357391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.357262    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-xks-ca\") pod \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\" (UID: \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\") "
Nov 02 15:12:11.357391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.357290    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-kube-api-access-8m5hs\" (UniqueName: \"kubernetes.io/projected/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-kube-api-access-8m5hs\") pod \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\" (UID: \"1a9a7b60-6e32-4f02-a984-f5e472cb7dea\") "
Nov 02 15:12:11.357549 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.357523    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-openshift-ca" pod "1a9a7b60-6e32-4f02-a984-f5e472cb7dea" (UID: "1a9a7b60-6e32-4f02-a984-f5e472cb7dea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:11.357735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.357716    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:11.359328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.359292    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-kube-api-access-8m5hs" pod "1a9a7b60-6e32-4f02-a984-f5e472cb7dea" (UID: "1a9a7b60-6e32-4f02-a984-f5e472cb7dea"). InnerVolumeSpecName "kube-api-access-8m5hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:11.359843 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.359822    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-xks-ca" pod "1a9a7b60-6e32-4f02-a984-f5e472cb7dea" (UID: "1a9a7b60-6e32-4f02-a984-f5e472cb7dea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:11.459017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.458932    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:11.459017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:11.458965    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m5hs\" (UniqueName: \"kubernetes.io/projected/1a9a7b60-6e32-4f02-a984-f5e472cb7dea-kube-api-access-8m5hs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:12.135733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:12.135691    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"
Nov 02 15:12:12.135733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:12.135706    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz" event={"ID":"1a9a7b60-6e32-4f02-a984-f5e472cb7dea","Type":"ContainerDied","Data":"22b30258920122de71b8d324f202cf31e2290bbf531e09e7792d3ba0b01170ae"}
Nov 02 15:12:12.135988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:12.135782    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22b30258920122de71b8d324f202cf31e2290bbf531e09e7792d3ba0b01170ae"
Nov 02 15:12:14.273142 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:12:14.273055    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:12:14.422460 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.422429    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"]
Nov 02 15:12:14.422907 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.422893    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1a9a7b60-6e32-4f02-a984-f5e472cb7dea" containerName="revoke-keys"
Nov 02 15:12:14.422951 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.422908    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9a7b60-6e32-4f02-a984-f5e472cb7dea" containerName="revoke-keys"
Nov 02 15:12:14.423002 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.422993    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9a7b60-6e32-4f02-a984-f5e472cb7dea" containerName="revoke-keys"
Nov 02 15:12:14.426570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.426546    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:14.486520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.486484    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"]
Nov 02 15:12:14.492632 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.492601    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99d039c-b6t7b\" (UID: \"1aeef115-33ea-4680-8e02-089853d0709f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:14.492632 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.492634    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99d039c-b6t7b\" (UID: \"1aeef115-33ea-4680-8e02-089853d0709f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:14.492880 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.492858    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2kqq\" (UniqueName: \"kubernetes.io/projected/1aeef115-33ea-4680-8e02-089853d0709f-kube-api-access-v2kqq\") pod \"maas-api-revoke-sub-f99d039c-b6t7b\" (UID: \"1aeef115-33ea-4680-8e02-089853d0709f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:14.531638 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.531564    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:12:14.531638 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.531597    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:12:14.531801 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.531660    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:12:14.531801 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.531689    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:12:14.587649 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.587621    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"]
Nov 02 15:12:14.591527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.591504    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:14.593362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.593340    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2kqq\" (UniqueName: \"kubernetes.io/projected/1aeef115-33ea-4680-8e02-089853d0709f-kube-api-access-v2kqq\") pod \"maas-api-revoke-sub-f99d039c-b6t7b\" (UID: \"1aeef115-33ea-4680-8e02-089853d0709f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:14.593486 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.593399    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99d039c-b6t7b\" (UID: \"1aeef115-33ea-4680-8e02-089853d0709f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:14.593486 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.593428    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99d039c-b6t7b\" (UID: \"1aeef115-33ea-4680-8e02-089853d0709f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:14.594424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.594399    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99d039c-b6t7b\" (UID: \"1aeef115-33ea-4680-8e02-089853d0709f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:14.595957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.595934    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99d039c-b6t7b\" (UID: \"1aeef115-33ea-4680-8e02-089853d0709f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:14.653234 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.649589    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"]
Nov 02 15:12:14.656285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.656256    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2kqq\" (UniqueName: \"kubernetes.io/projected/1aeef115-33ea-4680-8e02-089853d0709f-kube-api-access-v2kqq\") pod \"maas-api-revoke-sub-f99d039c-b6t7b\" (UID: \"1aeef115-33ea-4680-8e02-089853d0709f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:14.694454 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.694422    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzrc2\" (UniqueName: \"kubernetes.io/projected/0d323de6-da2b-4a24-a19f-7faa80858c6e-kube-api-access-nzrc2\") pod \"maas-api-revoke-sub-096d7a10-gbfmd\" (UID: \"0d323de6-da2b-4a24-a19f-7faa80858c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:14.694634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.694541    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-096d7a10-gbfmd\" (UID: \"0d323de6-da2b-4a24-a19f-7faa80858c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:14.694634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.694585    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-096d7a10-gbfmd\" (UID: \"0d323de6-da2b-4a24-a19f-7faa80858c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:14.741011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.740975    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:14.798960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.797005    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzrc2\" (UniqueName: \"kubernetes.io/projected/0d323de6-da2b-4a24-a19f-7faa80858c6e-kube-api-access-nzrc2\") pod \"maas-api-revoke-sub-096d7a10-gbfmd\" (UID: \"0d323de6-da2b-4a24-a19f-7faa80858c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:14.798960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.797114    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-096d7a10-gbfmd\" (UID: \"0d323de6-da2b-4a24-a19f-7faa80858c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:14.798960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.797171    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-096d7a10-gbfmd\" (UID: \"0d323de6-da2b-4a24-a19f-7faa80858c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:14.798960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.798055    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-096d7a10-gbfmd\" (UID: \"0d323de6-da2b-4a24-a19f-7faa80858c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:14.824340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.817338    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-096d7a10-gbfmd\" (UID: \"0d323de6-da2b-4a24-a19f-7faa80858c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:14.857954 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.857918    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzrc2\" (UniqueName: \"kubernetes.io/projected/0d323de6-da2b-4a24-a19f-7faa80858c6e-kube-api-access-nzrc2\") pod \"maas-api-revoke-sub-096d7a10-gbfmd\" (UID: \"0d323de6-da2b-4a24-a19f-7faa80858c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:14.914588 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:14.914554    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:15.119551 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.119518    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"]
Nov 02 15:12:15.120393 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:15.120364    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aeef115_33ea_4680_8e02_089853d0709f.slice/crio-76ce7218ae3ebd9a2c099da702cbcc9fa4e54f5a2335cd46f41a799333a36dd9 WatchSource:0}: Error finding container 76ce7218ae3ebd9a2c099da702cbcc9fa4e54f5a2335cd46f41a799333a36dd9: Status 404 returned error can't find the container with id 76ce7218ae3ebd9a2c099da702cbcc9fa4e54f5a2335cd46f41a799333a36dd9
Nov 02 15:12:15.155611 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.155583    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b" event={"ID":"1aeef115-33ea-4680-8e02-089853d0709f","Type":"ContainerStarted","Data":"76ce7218ae3ebd9a2c099da702cbcc9fa4e54f5a2335cd46f41a799333a36dd9"}
Nov 02 15:12:15.293687 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.293660    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"]
Nov 02 15:12:15.294983 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:15.294953    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d323de6_da2b_4a24_a19f_7faa80858c6e.slice/crio-48577e4fd8882534e8fad8e0cd8ef08e07af707e7322f626f62fa130b976c1bc WatchSource:0}: Error finding container 48577e4fd8882534e8fad8e0cd8ef08e07af707e7322f626f62fa130b976c1bc: Status 404 returned error can't find the container with id 48577e4fd8882534e8fad8e0cd8ef08e07af707e7322f626f62fa130b976c1bc
Nov 02 15:12:15.445800 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.445728    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"]
Nov 02 15:12:15.450874 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.450851    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:15.505306 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.505249    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ec3d0f8-tkc9m\" (UID: \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:15.505795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.505398    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfvsz\" (UniqueName: \"kubernetes.io/projected/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-kube-api-access-wfvsz\") pod \"maas-api-revoke-sub-4ec3d0f8-tkc9m\" (UID: \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:15.505795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.505545    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ec3d0f8-tkc9m\" (UID: \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:15.510298 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.510275    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"]
Nov 02 15:12:15.606992 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.606953    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfvsz\" (UniqueName: \"kubernetes.io/projected/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-kube-api-access-wfvsz\") pod \"maas-api-revoke-sub-4ec3d0f8-tkc9m\" (UID: \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:15.607164 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.607095    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ec3d0f8-tkc9m\" (UID: \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:15.607294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.607260    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ec3d0f8-tkc9m\" (UID: \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:15.608037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.608010    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ec3d0f8-tkc9m\" (UID: \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:15.609826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.609805    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ec3d0f8-tkc9m\" (UID: \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:15.668504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.668473    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfvsz\" (UniqueName: \"kubernetes.io/projected/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-kube-api-access-wfvsz\") pod \"maas-api-revoke-sub-4ec3d0f8-tkc9m\" (UID: \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:15.769360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:15.769331    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:16.151190 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:16.151162    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"]
Nov 02 15:12:16.164575 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:16.164543    4891 generic.go:356] "Generic (PLEG): container finished" podID="0d323de6-da2b-4a24-a19f-7faa80858c6e" containerID="53adb112e663dbaca3e1a2ea8fad9c6622261690f9bc2d3f407aac8d0faddaae" exitCode=0
Nov 02 15:12:16.164686 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:16.164626    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd" event={"ID":"0d323de6-da2b-4a24-a19f-7faa80858c6e","Type":"ContainerDied","Data":"53adb112e663dbaca3e1a2ea8fad9c6622261690f9bc2d3f407aac8d0faddaae"}
Nov 02 15:12:16.164724 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:16.164688    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd" event={"ID":"0d323de6-da2b-4a24-a19f-7faa80858c6e","Type":"ContainerStarted","Data":"48577e4fd8882534e8fad8e0cd8ef08e07af707e7322f626f62fa130b976c1bc"}
Nov 02 15:12:16.166280 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:16.166261    4891 generic.go:356] "Generic (PLEG): container finished" podID="1aeef115-33ea-4680-8e02-089853d0709f" containerID="255b8a5d93c0dfad97056f15c3037db95c88f0a79efb9fec72b0a759afe3e7d3" exitCode=0
Nov 02 15:12:16.166398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:16.166348    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b" event={"ID":"1aeef115-33ea-4680-8e02-089853d0709f","Type":"ContainerDied","Data":"255b8a5d93c0dfad97056f15c3037db95c88f0a79efb9fec72b0a759afe3e7d3"}
Nov 02 15:12:16.175463 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:16.175437    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d844a2a_cfa1_4a8c_8b42_4c33acbcceb6.slice/crio-ecdda541644ee7f56271523c081d7ae0126e36deed23e72824c77f3150c4c204 WatchSource:0}: Error finding container ecdda541644ee7f56271523c081d7ae0126e36deed23e72824c77f3150c4c204: Status 404 returned error can't find the container with id ecdda541644ee7f56271523c081d7ae0126e36deed23e72824c77f3150c4c204
Nov 02 15:12:17.175048 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.175017    4891 generic.go:356] "Generic (PLEG): container finished" podID="0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6" containerID="ec69b803ebe193f27a5eedb9d1b3fd7fd85a5ff42df1e135fd4951a1c6ec0779" exitCode=0
Nov 02 15:12:17.175527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.175115    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m" event={"ID":"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6","Type":"ContainerDied","Data":"ec69b803ebe193f27a5eedb9d1b3fd7fd85a5ff42df1e135fd4951a1c6ec0779"}
Nov 02 15:12:17.175527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.175177    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m" event={"ID":"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6","Type":"ContainerStarted","Data":"ecdda541644ee7f56271523c081d7ae0126e36deed23e72824c77f3150c4c204"}
Nov 02 15:12:17.332472 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.332445    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:17.355302 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.355279    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:17.433062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.433032    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-xks-ca\") pod \"1aeef115-33ea-4680-8e02-089853d0709f\" (UID: \"1aeef115-33ea-4680-8e02-089853d0709f\") "
Nov 02 15:12:17.433236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.433117    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aeef115-33ea-4680-8e02-089853d0709f-kube-api-access-v2kqq\" (UniqueName: \"kubernetes.io/projected/1aeef115-33ea-4680-8e02-089853d0709f-kube-api-access-v2kqq\") pod \"1aeef115-33ea-4680-8e02-089853d0709f\" (UID: \"1aeef115-33ea-4680-8e02-089853d0709f\") "
Nov 02 15:12:17.433236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.433141    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-openshift-ca\") pod \"0d323de6-da2b-4a24-a19f-7faa80858c6e\" (UID: \"0d323de6-da2b-4a24-a19f-7faa80858c6e\") "
Nov 02 15:12:17.433236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.433162    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d323de6-da2b-4a24-a19f-7faa80858c6e-kube-api-access-nzrc2\" (UniqueName: \"kubernetes.io/projected/0d323de6-da2b-4a24-a19f-7faa80858c6e-kube-api-access-nzrc2\") pod \"0d323de6-da2b-4a24-a19f-7faa80858c6e\" (UID: \"0d323de6-da2b-4a24-a19f-7faa80858c6e\") "
Nov 02 15:12:17.433236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.433189    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-openshift-ca\") pod \"1aeef115-33ea-4680-8e02-089853d0709f\" (UID: \"1aeef115-33ea-4680-8e02-089853d0709f\") "
Nov 02 15:12:17.433466 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.433337    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-xks-ca\") pod \"0d323de6-da2b-4a24-a19f-7faa80858c6e\" (UID: \"0d323de6-da2b-4a24-a19f-7faa80858c6e\") "
Nov 02 15:12:17.433607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.433580    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-openshift-ca" pod "0d323de6-da2b-4a24-a19f-7faa80858c6e" (UID: "0d323de6-da2b-4a24-a19f-7faa80858c6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:17.433607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.433589    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-openshift-ca" pod "1aeef115-33ea-4680-8e02-089853d0709f" (UID: "1aeef115-33ea-4680-8e02-089853d0709f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:17.433845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.433822    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:17.433897 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.433850    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:17.435321 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.435280    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-xks-ca" pod "1aeef115-33ea-4680-8e02-089853d0709f" (UID: "1aeef115-33ea-4680-8e02-089853d0709f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:17.435401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.435363    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d323de6-da2b-4a24-a19f-7faa80858c6e-kube-api-access-nzrc2" pod "0d323de6-da2b-4a24-a19f-7faa80858c6e" (UID: "0d323de6-da2b-4a24-a19f-7faa80858c6e"). InnerVolumeSpecName "kube-api-access-nzrc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:17.435681 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.435654    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-xks-ca" pod "0d323de6-da2b-4a24-a19f-7faa80858c6e" (UID: "0d323de6-da2b-4a24-a19f-7faa80858c6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:17.435768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.435654    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aeef115-33ea-4680-8e02-089853d0709f-kube-api-access-v2kqq" pod "1aeef115-33ea-4680-8e02-089853d0709f" (UID: "1aeef115-33ea-4680-8e02-089853d0709f"). InnerVolumeSpecName "kube-api-access-v2kqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:17.534384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.534345    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2kqq\" (UniqueName: \"kubernetes.io/projected/1aeef115-33ea-4680-8e02-089853d0709f-kube-api-access-v2kqq\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:17.534384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.534382    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzrc2\" (UniqueName: \"kubernetes.io/projected/0d323de6-da2b-4a24-a19f-7faa80858c6e-kube-api-access-nzrc2\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:17.534589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.534395    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d323de6-da2b-4a24-a19f-7faa80858c6e-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:17.534589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:17.534409    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aeef115-33ea-4680-8e02-089853d0709f-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:18.191020 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.190986    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"
Nov 02 15:12:18.191020 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.191010    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd" event={"ID":"0d323de6-da2b-4a24-a19f-7faa80858c6e","Type":"ContainerDied","Data":"48577e4fd8882534e8fad8e0cd8ef08e07af707e7322f626f62fa130b976c1bc"}
Nov 02 15:12:18.191487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.191079    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48577e4fd8882534e8fad8e0cd8ef08e07af707e7322f626f62fa130b976c1bc"
Nov 02 15:12:18.192627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.192596    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b" event={"ID":"1aeef115-33ea-4680-8e02-089853d0709f","Type":"ContainerDied","Data":"76ce7218ae3ebd9a2c099da702cbcc9fa4e54f5a2335cd46f41a799333a36dd9"}
Nov 02 15:12:18.192752 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.192648    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76ce7218ae3ebd9a2c099da702cbcc9fa4e54f5a2335cd46f41a799333a36dd9"
Nov 02 15:12:18.192752 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.192722    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"
Nov 02 15:12:18.348557 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.348532    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:18.443100 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.443028    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-xks-ca\") pod \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\" (UID: \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\") "
Nov 02 15:12:18.443237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.443130    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-kube-api-access-wfvsz\" (UniqueName: \"kubernetes.io/projected/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-kube-api-access-wfvsz\") pod \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\" (UID: \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\") "
Nov 02 15:12:18.443237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.443202    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-openshift-ca\") pod \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\" (UID: \"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6\") "
Nov 02 15:12:18.443540 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.443510    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-openshift-ca" pod "0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6" (UID: "0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:18.445087 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.445059    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-xks-ca" pod "0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6" (UID: "0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:18.445221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.445201    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-kube-api-access-wfvsz" pod "0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6" (UID: "0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6"). InnerVolumeSpecName "kube-api-access-wfvsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:18.544040 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.544011    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:18.544040 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.544040    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:18.544182 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.544055    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfvsz\" (UniqueName: \"kubernetes.io/projected/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6-kube-api-access-wfvsz\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:18.566823 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.566793    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"]
Nov 02 15:12:18.567492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.567472    4891 cpu_manager.go:414] "containerMap: removing container" podUID="0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6" containerName="revoke-keys"
Nov 02 15:12:18.567552 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.567497    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6" containerName="revoke-keys"
Nov 02 15:12:18.567590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.567554    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1aeef115-33ea-4680-8e02-089853d0709f" containerName="revoke-keys"
Nov 02 15:12:18.567590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.567562    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aeef115-33ea-4680-8e02-089853d0709f" containerName="revoke-keys"
Nov 02 15:12:18.567590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.567575    4891 cpu_manager.go:414] "containerMap: removing container" podUID="0d323de6-da2b-4a24-a19f-7faa80858c6e" containerName="revoke-keys"
Nov 02 15:12:18.567590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.567582    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d323de6-da2b-4a24-a19f-7faa80858c6e" containerName="revoke-keys"
Nov 02 15:12:18.567753 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.567722    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aeef115-33ea-4680-8e02-089853d0709f" containerName="revoke-keys"
Nov 02 15:12:18.567753 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.567735    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d323de6-da2b-4a24-a19f-7faa80858c6e" containerName="revoke-keys"
Nov 02 15:12:18.567753 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.567750    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6" containerName="revoke-keys"
Nov 02 15:12:18.571037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.571017    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:18.628740 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.628711    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"]
Nov 02 15:12:18.644871 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.644841    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr729\" (UniqueName: \"kubernetes.io/projected/ab3c0aa3-cf80-4687-a90d-48547900c30c-kube-api-access-rr729\") pod \"maas-api-revoke-sub-b196c434-l9qk9\" (UID: \"ab3c0aa3-cf80-4687-a90d-48547900c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:18.645045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.644876    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b196c434-l9qk9\" (UID: \"ab3c0aa3-cf80-4687-a90d-48547900c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:18.645110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.645079    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b196c434-l9qk9\" (UID: \"ab3c0aa3-cf80-4687-a90d-48547900c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:18.745922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.745832    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b196c434-l9qk9\" (UID: \"ab3c0aa3-cf80-4687-a90d-48547900c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:18.746106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.745946    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr729\" (UniqueName: \"kubernetes.io/projected/ab3c0aa3-cf80-4687-a90d-48547900c30c-kube-api-access-rr729\") pod \"maas-api-revoke-sub-b196c434-l9qk9\" (UID: \"ab3c0aa3-cf80-4687-a90d-48547900c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:18.746106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.745973    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b196c434-l9qk9\" (UID: \"ab3c0aa3-cf80-4687-a90d-48547900c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:18.746717 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.746696    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b196c434-l9qk9\" (UID: \"ab3c0aa3-cf80-4687-a90d-48547900c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:18.748318 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.748274    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b196c434-l9qk9\" (UID: \"ab3c0aa3-cf80-4687-a90d-48547900c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:18.807140 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.807107    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr729\" (UniqueName: \"kubernetes.io/projected/ab3c0aa3-cf80-4687-a90d-48547900c30c-kube-api-access-rr729\") pod \"maas-api-revoke-sub-b196c434-l9qk9\" (UID: \"ab3c0aa3-cf80-4687-a90d-48547900c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:18.885091 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:18.885056    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:19.034808 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.034775    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"]
Nov 02 15:12:19.039400 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.039384    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.093660 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.093628    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6dc69b-openshift-default-dockercfg-pbd2c\""
Nov 02 15:12:19.098301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.098269    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"]
Nov 02 15:12:19.149394 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.149360    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9fnx\" (UniqueName: \"kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-kube-api-access-f9fnx\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.149546 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.149445    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.149546 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.149472    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.149546 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.149509    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806a03af-da93-4064-a70f-cc3ee68e5c9d-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.149649 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.149602    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.149649 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.149631    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.149716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.149650    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.149748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.149732    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.149784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.149767    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.199906 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.199882    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"
Nov 02 15:12:19.200279 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.199881    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m" event={"ID":"0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6","Type":"ContainerDied","Data":"ecdda541644ee7f56271523c081d7ae0126e36deed23e72824c77f3150c4c204"}
Nov 02 15:12:19.200279 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.200006    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecdda541644ee7f56271523c081d7ae0126e36deed23e72824c77f3150c4c204"
Nov 02 15:12:19.251089 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251060    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.251250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251101    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.251250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251158    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9fnx\" (UniqueName: \"kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-kube-api-access-f9fnx\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.251250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251178    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.251250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251196    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.251499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251363    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806a03af-da93-4064-a70f-cc3ee68e5c9d-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.251499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251444    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.251499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251479    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.251657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251506    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.251657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251550    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.251858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251834    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.251931 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251853    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.251977 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.251930    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.252140 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.252114    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806a03af-da93-4064-a70f-cc3ee68e5c9d-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.253562 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.253537    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.253722 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.253705    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.262380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.262361    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"]
Nov 02 15:12:19.263209 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:19.263174    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3c0aa3_cf80_4687_a90d_48547900c30c.slice/crio-745293199a0986cb424d535dbc83091aec7c3d25285eb52712a9580ac8789fdc WatchSource:0}: Error finding container 745293199a0986cb424d535dbc83091aec7c3d25285eb52712a9580ac8789fdc: Status 404 returned error can't find the container with id 745293199a0986cb424d535dbc83091aec7c3d25285eb52712a9580ac8789fdc
Nov 02 15:12:19.312015 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.311985    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.312182 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.312164    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9fnx\" (UniqueName: \"kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-kube-api-access-f9fnx\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.354252 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.354219    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:19.734873 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.734831    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"]
Nov 02 15:12:19.772000 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:19.771969    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806a03af_da93_4064_a70f_cc3ee68e5c9d.slice/crio-92ed873a603437829b717ee2093e7e5bf1199d04c0f519ccf9cce679336e0992 WatchSource:0}: Error finding container 92ed873a603437829b717ee2093e7e5bf1199d04c0f519ccf9cce679336e0992: Status 404 returned error can't find the container with id 92ed873a603437829b717ee2093e7e5bf1199d04c0f519ccf9cce679336e0992
Nov 02 15:12:19.774884 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.774853    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:19.774959 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.774919    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:19.774959 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:19.774949    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:20.208392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:20.208282    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" event={"ID":"806a03af-da93-4064-a70f-cc3ee68e5c9d","Type":"ContainerStarted","Data":"4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1"}
Nov 02 15:12:20.208392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:20.208368    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" event={"ID":"806a03af-da93-4064-a70f-cc3ee68e5c9d","Type":"ContainerStarted","Data":"92ed873a603437829b717ee2093e7e5bf1199d04c0f519ccf9cce679336e0992"}
Nov 02 15:12:20.209966 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:20.209941    4891 generic.go:356] "Generic (PLEG): container finished" podID="ab3c0aa3-cf80-4687-a90d-48547900c30c" containerID="75d7ce1653667d0bf5a8ed974473bbfca2247655e503af2381147a19d13fb929" exitCode=0
Nov 02 15:12:20.210070 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:20.210012    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9" event={"ID":"ab3c0aa3-cf80-4687-a90d-48547900c30c","Type":"ContainerDied","Data":"75d7ce1653667d0bf5a8ed974473bbfca2247655e503af2381147a19d13fb929"}
Nov 02 15:12:20.210070 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:20.210043    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9" event={"ID":"ab3c0aa3-cf80-4687-a90d-48547900c30c","Type":"ContainerStarted","Data":"745293199a0986cb424d535dbc83091aec7c3d25285eb52712a9580ac8789fdc"}
Nov 02 15:12:20.330736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:20.330678    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" podStartSLOduration=2.330663354 podStartE2EDuration="2.330663354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:20.328134818 +0000 UTC m=+947.920751629" watchObservedRunningTime="2026-11-02 15:12:20.330663354 +0000 UTC m=+947.923280150"
Nov 02 15:12:20.355082 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:20.355049    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:20.356508 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:20.356487    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:20.356581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:20.356531    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 15:12:21.370667 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:21.370629    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:21.371138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:21.370689    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 15:12:21.401045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:21.401020    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:21.476032 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:21.475939    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-xks-ca\") pod \"ab3c0aa3-cf80-4687-a90d-48547900c30c\" (UID: \"ab3c0aa3-cf80-4687-a90d-48547900c30c\") "
Nov 02 15:12:21.476032 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:21.476030    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-openshift-ca\") pod \"ab3c0aa3-cf80-4687-a90d-48547900c30c\" (UID: \"ab3c0aa3-cf80-4687-a90d-48547900c30c\") "
Nov 02 15:12:21.476235 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:21.476124    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3c0aa3-cf80-4687-a90d-48547900c30c-kube-api-access-rr729\" (UniqueName: \"kubernetes.io/projected/ab3c0aa3-cf80-4687-a90d-48547900c30c-kube-api-access-rr729\") pod \"ab3c0aa3-cf80-4687-a90d-48547900c30c\" (UID: \"ab3c0aa3-cf80-4687-a90d-48547900c30c\") "
Nov 02 15:12:21.476414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:21.476385    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-openshift-ca" pod "ab3c0aa3-cf80-4687-a90d-48547900c30c" (UID: "ab3c0aa3-cf80-4687-a90d-48547900c30c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:21.478125 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:21.478100    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-xks-ca" pod "ab3c0aa3-cf80-4687-a90d-48547900c30c" (UID: "ab3c0aa3-cf80-4687-a90d-48547900c30c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:21.478211 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:21.478141    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3c0aa3-cf80-4687-a90d-48547900c30c-kube-api-access-rr729" pod "ab3c0aa3-cf80-4687-a90d-48547900c30c" (UID: "ab3c0aa3-cf80-4687-a90d-48547900c30c"). InnerVolumeSpecName "kube-api-access-rr729". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:21.577330 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:21.577287    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:21.577330 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:21.577331    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3c0aa3-cf80-4687-a90d-48547900c30c-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:21.577503 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:21.577342    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr729\" (UniqueName: \"kubernetes.io/projected/ab3c0aa3-cf80-4687-a90d-48547900c30c-kube-api-access-rr729\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:22.224464 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:22.224426    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"
Nov 02 15:12:22.224464 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:22.224458    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9" event={"ID":"ab3c0aa3-cf80-4687-a90d-48547900c30c","Type":"ContainerDied","Data":"745293199a0986cb424d535dbc83091aec7c3d25285eb52712a9580ac8789fdc"}
Nov 02 15:12:22.224719 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:22.224503    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="745293199a0986cb424d535dbc83091aec7c3d25285eb52712a9580ac8789fdc"
Nov 02 15:12:22.362822 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:22.362777    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:22.363172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:22.363140    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 15:12:23.357746 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:23.357705    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:23.358390 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:23.358353    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 15:12:24.354708 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.354669    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:24.354979 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.354952    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 15:12:24.676802 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.676703    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"]
Nov 02 15:12:24.677975 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.677947    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ab3c0aa3-cf80-4687-a90d-48547900c30c" containerName="revoke-keys"
Nov 02 15:12:24.678114 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.678100    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab3c0aa3-cf80-4687-a90d-48547900c30c" containerName="revoke-keys"
Nov 02 15:12:24.678406 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.678390    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab3c0aa3-cf80-4687-a90d-48547900c30c" containerName="revoke-keys"
Nov 02 15:12:24.688888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.688846    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:24.726665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.726632    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"]
Nov 02 15:12:24.773335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.772700    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:12:24.773537 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.773432    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:12:24.773537 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.773516    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:12:24.795565 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.795532    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:12:24.816000 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.815965    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mwtf\" (UniqueName: \"kubernetes.io/projected/819d0f4a-890b-4fcf-8059-dfbd30e77644-kube-api-access-7mwtf\") pod \"maas-api-revoke-sub-679d8220-2q87c\" (UID: \"819d0f4a-890b-4fcf-8059-dfbd30e77644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:24.816186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.816097    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-679d8220-2q87c\" (UID: \"819d0f4a-890b-4fcf-8059-dfbd30e77644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:24.816186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.816121    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-679d8220-2q87c\" (UID: \"819d0f4a-890b-4fcf-8059-dfbd30e77644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:24.916903 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.916863    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mwtf\" (UniqueName: \"kubernetes.io/projected/819d0f4a-890b-4fcf-8059-dfbd30e77644-kube-api-access-7mwtf\") pod \"maas-api-revoke-sub-679d8220-2q87c\" (UID: \"819d0f4a-890b-4fcf-8059-dfbd30e77644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:24.917093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.916956    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-679d8220-2q87c\" (UID: \"819d0f4a-890b-4fcf-8059-dfbd30e77644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:24.917093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.916976    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-679d8220-2q87c\" (UID: \"819d0f4a-890b-4fcf-8059-dfbd30e77644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:24.917698 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.917675    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-679d8220-2q87c\" (UID: \"819d0f4a-890b-4fcf-8059-dfbd30e77644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:24.919424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.919403    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-679d8220-2q87c\" (UID: \"819d0f4a-890b-4fcf-8059-dfbd30e77644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:24.995726 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:24.995644    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mwtf\" (UniqueName: \"kubernetes.io/projected/819d0f4a-890b-4fcf-8059-dfbd30e77644-kube-api-access-7mwtf\") pod \"maas-api-revoke-sub-679d8220-2q87c\" (UID: \"819d0f4a-890b-4fcf-8059-dfbd30e77644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:25.015216 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:25.015189    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:25.356757 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:25.356716    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:25.357026 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:25.356987    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 15:12:25.431449 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:25.430168    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"]
Nov 02 15:12:25.431449 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:25.430883    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819d0f4a_890b_4fcf_8059_dfbd30e77644.slice/crio-8419da4b711f5227fcb9a069183d540c9944b1b28bb1a5c081cc67a997369f21 WatchSource:0}: Error finding container 8419da4b711f5227fcb9a069183d540c9944b1b28bb1a5c081cc67a997369f21: Status 404 returned error can't find the container with id 8419da4b711f5227fcb9a069183d540c9944b1b28bb1a5c081cc67a997369f21
Nov 02 15:12:26.286042 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:26.286007    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c" event={"ID":"819d0f4a-890b-4fcf-8059-dfbd30e77644","Type":"ContainerStarted","Data":"e24cf6c3bb2c3f13710fa0805cc81c59eddccd82fc5f01780442443256d62c6c"}
Nov 02 15:12:26.286551 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:26.286062    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c" event={"ID":"819d0f4a-890b-4fcf-8059-dfbd30e77644","Type":"ContainerStarted","Data":"8419da4b711f5227fcb9a069183d540c9944b1b28bb1a5c081cc67a997369f21"}
Nov 02 15:12:26.355038 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:26.355006    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:26.355204 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:26.355065    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 02 15:12:26.404199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:26.404133    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c" podStartSLOduration=2.404113993 podStartE2EDuration="2.404113993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:26.402026176 +0000 UTC m=+953.994642976" watchObservedRunningTime="2026-11-02 15:12:26.404113993 +0000 UTC m=+953.996730793"
Nov 02 15:12:27.293874 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:27.293840    4891 generic.go:356] "Generic (PLEG): container finished" podID="819d0f4a-890b-4fcf-8059-dfbd30e77644" containerID="e24cf6c3bb2c3f13710fa0805cc81c59eddccd82fc5f01780442443256d62c6c" exitCode=0
Nov 02 15:12:27.294286 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:27.293909    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c" event={"ID":"819d0f4a-890b-4fcf-8059-dfbd30e77644","Type":"ContainerDied","Data":"e24cf6c3bb2c3f13710fa0805cc81c59eddccd82fc5f01780442443256d62c6c"}
Nov 02 15:12:27.359442 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:27.359412    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:27.359712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:27.359685    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:27.360362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:27.360346    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:12:28.448468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:28.448443    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:28.557233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:28.557153    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819d0f4a-890b-4fcf-8059-dfbd30e77644-kube-api-access-7mwtf\" (UniqueName: \"kubernetes.io/projected/819d0f4a-890b-4fcf-8059-dfbd30e77644-kube-api-access-7mwtf\") pod \"819d0f4a-890b-4fcf-8059-dfbd30e77644\" (UID: \"819d0f4a-890b-4fcf-8059-dfbd30e77644\") "
Nov 02 15:12:28.557233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:28.557229    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-xks-ca\") pod \"819d0f4a-890b-4fcf-8059-dfbd30e77644\" (UID: \"819d0f4a-890b-4fcf-8059-dfbd30e77644\") "
Nov 02 15:12:28.557466 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:28.557370    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-openshift-ca\") pod \"819d0f4a-890b-4fcf-8059-dfbd30e77644\" (UID: \"819d0f4a-890b-4fcf-8059-dfbd30e77644\") "
Nov 02 15:12:28.557704 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:28.557679    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-openshift-ca" pod "819d0f4a-890b-4fcf-8059-dfbd30e77644" (UID: "819d0f4a-890b-4fcf-8059-dfbd30e77644"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:28.557859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:28.557842    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:28.559272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:28.559244    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-xks-ca" pod "819d0f4a-890b-4fcf-8059-dfbd30e77644" (UID: "819d0f4a-890b-4fcf-8059-dfbd30e77644"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:28.559398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:28.559352    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819d0f4a-890b-4fcf-8059-dfbd30e77644-kube-api-access-7mwtf" pod "819d0f4a-890b-4fcf-8059-dfbd30e77644" (UID: "819d0f4a-890b-4fcf-8059-dfbd30e77644"). InnerVolumeSpecName "kube-api-access-7mwtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:28.659597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:28.659558    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mwtf\" (UniqueName: \"kubernetes.io/projected/819d0f4a-890b-4fcf-8059-dfbd30e77644-kube-api-access-7mwtf\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:28.659597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:28.659596    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819d0f4a-890b-4fcf-8059-dfbd30e77644-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:29.308946 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:29.308919    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"
Nov 02 15:12:29.309099 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:29.308920    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c" event={"ID":"819d0f4a-890b-4fcf-8059-dfbd30e77644","Type":"ContainerDied","Data":"8419da4b711f5227fcb9a069183d540c9944b1b28bb1a5c081cc67a997369f21"}
Nov 02 15:12:29.309099 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:29.309065    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8419da4b711f5227fcb9a069183d540c9944b1b28bb1a5c081cc67a997369f21"
Nov 02 15:12:34.622726 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.622694    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"]
Nov 02 15:12:34.623134 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.623121    4891 cpu_manager.go:414] "containerMap: removing container" podUID="819d0f4a-890b-4fcf-8059-dfbd30e77644" containerName="revoke-keys"
Nov 02 15:12:34.623184 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.623134    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819d0f4a-890b-4fcf-8059-dfbd30e77644" containerName="revoke-keys"
Nov 02 15:12:34.623279 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.623269    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="819d0f4a-890b-4fcf-8059-dfbd30e77644" containerName="revoke-keys"
Nov 02 15:12:34.627720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.627705    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:34.717353 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.717293    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28dd6acc-9w25x\" (UID: \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:34.717353 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.717357    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4468\" (UniqueName: \"kubernetes.io/projected/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-kube-api-access-v4468\") pod \"maas-api-revoke-sub-28dd6acc-9w25x\" (UID: \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:34.717581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.717519    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28dd6acc-9w25x\" (UID: \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:34.732436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.732397    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:12:34.732436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.732405    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:12:34.732436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.732408    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:12:34.732639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.732445    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:12:34.736068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.736048    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"]
Nov 02 15:12:34.818687 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.818656    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28dd6acc-9w25x\" (UID: \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:34.818843 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.818765    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28dd6acc-9w25x\" (UID: \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:34.818843 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.818796    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4468\" (UniqueName: \"kubernetes.io/projected/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-kube-api-access-v4468\") pod \"maas-api-revoke-sub-28dd6acc-9w25x\" (UID: \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:34.819306 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.819288    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28dd6acc-9w25x\" (UID: \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:34.821036 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.821008    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28dd6acc-9w25x\" (UID: \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:34.881886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.881814    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4468\" (UniqueName: \"kubernetes.io/projected/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-kube-api-access-v4468\") pod \"maas-api-revoke-sub-28dd6acc-9w25x\" (UID: \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:34.941042 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:34.940936    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:35.321599 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:35.321569    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"]
Nov 02 15:12:35.322504 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:35.322473    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5726fda9_7b7b_4170_a62b_252d7f1d9ebd.slice/crio-0f48df45abc9022c626e3a7e873a965d4e2f50a9863be3906b5cd42af393b2cc WatchSource:0}: Error finding container 0f48df45abc9022c626e3a7e873a965d4e2f50a9863be3906b5cd42af393b2cc: Status 404 returned error can't find the container with id 0f48df45abc9022c626e3a7e873a965d4e2f50a9863be3906b5cd42af393b2cc
Nov 02 15:12:35.350188 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:35.350162    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x" event={"ID":"5726fda9-7b7b-4170-a62b-252d7f1d9ebd","Type":"ContainerStarted","Data":"0f48df45abc9022c626e3a7e873a965d4e2f50a9863be3906b5cd42af393b2cc"}
Nov 02 15:12:36.358660 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.358619    4891 generic.go:356] "Generic (PLEG): container finished" podID="5726fda9-7b7b-4170-a62b-252d7f1d9ebd" containerID="bfd328287ba990ef091c29e55a9d1e9b239e655428a2af764b425b50f149132b" exitCode=0
Nov 02 15:12:36.359108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.358700    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x" event={"ID":"5726fda9-7b7b-4170-a62b-252d7f1d9ebd","Type":"ContainerDied","Data":"bfd328287ba990ef091c29e55a9d1e9b239e655428a2af764b425b50f149132b"}
Nov 02 15:12:36.796273 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.796234    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"]
Nov 02 15:12:36.800923 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.800898    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:36.909823 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.909786    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"]
Nov 02 15:12:36.940602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.940559    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13432aa0-5907-4e63-8d12-cacad202b86b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:36.940764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.940622    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:36.940764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.940737    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13432aa0-5907-4e63-8d12-cacad202b86b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:36.940831 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.940778    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:36.940889 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.940869    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:36.940922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.940904    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:36.940965 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.940947    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:36.941068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.941045    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:36.941142 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:36.941122    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxcmb\" (UniqueName: \"kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-kube-api-access-bxcmb\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.042125 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042093    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13432aa0-5907-4e63-8d12-cacad202b86b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.042334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042132    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.042334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042157    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13432aa0-5907-4e63-8d12-cacad202b86b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.042334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042175    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.042334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042202    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.042334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042221    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.042625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042386    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.042625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042433    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.042625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042495    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxcmb\" (UniqueName: \"kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-kube-api-access-bxcmb\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.042762 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042694    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.042906 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042875    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.042906 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042896    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.043031 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.042926    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.043031 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.043016    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13432aa0-5907-4e63-8d12-cacad202b86b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.044705 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.044686    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.044853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.044836    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13432aa0-5907-4e63-8d12-cacad202b86b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.103795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.103735    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.105252 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.105231    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxcmb\" (UniqueName: \"kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-kube-api-access-bxcmb\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qg548\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.115132 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.115103    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:37.513648 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.513624    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:37.564143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.564107    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"]
Nov 02 15:12:37.571952 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:37.571924    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13432aa0_5907_4e63_8d12_cacad202b86b.slice/crio-5a0c2ce1b864619440c6a942bbbc3d7d776a4e84d852bef29543fcb43ce65325 WatchSource:0}: Error finding container 5a0c2ce1b864619440c6a942bbbc3d7d776a4e84d852bef29543fcb43ce65325: Status 404 returned error can't find the container with id 5a0c2ce1b864619440c6a942bbbc3d7d776a4e84d852bef29543fcb43ce65325
Nov 02 15:12:37.574360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.574321    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:37.574438 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.574397    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:37.574438 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.574426    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:37.649413 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.649374    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-xks-ca\") pod \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\" (UID: \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\") "
Nov 02 15:12:37.649581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.649484    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-openshift-ca\") pod \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\" (UID: \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\") "
Nov 02 15:12:37.649648 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.649614    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-kube-api-access-v4468\" (UniqueName: \"kubernetes.io/projected/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-kube-api-access-v4468\") pod \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\" (UID: \"5726fda9-7b7b-4170-a62b-252d7f1d9ebd\") "
Nov 02 15:12:37.650199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.650144    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-openshift-ca" pod "5726fda9-7b7b-4170-a62b-252d7f1d9ebd" (UID: "5726fda9-7b7b-4170-a62b-252d7f1d9ebd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:37.652149 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.652119    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-kube-api-access-v4468" pod "5726fda9-7b7b-4170-a62b-252d7f1d9ebd" (UID: "5726fda9-7b7b-4170-a62b-252d7f1d9ebd"). InnerVolumeSpecName "kube-api-access-v4468". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:37.652250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.652230    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-xks-ca" pod "5726fda9-7b7b-4170-a62b-252d7f1d9ebd" (UID: "5726fda9-7b7b-4170-a62b-252d7f1d9ebd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:37.750949 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.750915    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4468\" (UniqueName: \"kubernetes.io/projected/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-kube-api-access-v4468\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:37.750949 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.750947    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:37.750949 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:37.750958    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5726fda9-7b7b-4170-a62b-252d7f1d9ebd-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:38.379260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:38.379165    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" event={"ID":"13432aa0-5907-4e63-8d12-cacad202b86b","Type":"ContainerStarted","Data":"993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757"}
Nov 02 15:12:38.379260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:38.379219    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" event={"ID":"13432aa0-5907-4e63-8d12-cacad202b86b","Type":"ContainerStarted","Data":"5a0c2ce1b864619440c6a942bbbc3d7d776a4e84d852bef29543fcb43ce65325"}
Nov 02 15:12:38.380748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:38.380728    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"
Nov 02 15:12:38.380844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:38.380756    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x" event={"ID":"5726fda9-7b7b-4170-a62b-252d7f1d9ebd","Type":"ContainerDied","Data":"0f48df45abc9022c626e3a7e873a965d4e2f50a9863be3906b5cd42af393b2cc"}
Nov 02 15:12:38.380844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:38.380793    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f48df45abc9022c626e3a7e873a965d4e2f50a9863be3906b5cd42af393b2cc"
Nov 02 15:12:38.604608 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:38.604531    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" podStartSLOduration=2.604506214 podStartE2EDuration="2.604506214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:38.549762774 +0000 UTC m=+966.142379576" watchObservedRunningTime="2026-11-02 15:12:38.604506214 +0000 UTC m=+966.197123015"
Nov 02 15:12:39.115488 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:39.115453    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:39.116785 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:39.116763    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qg548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:12:39.116848 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:39.116809    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" podUID="13432aa0-5907-4e63-8d12-cacad202b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 15:12:40.116084 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:40.116046    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qg548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:12:40.116524 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:40.116108    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" podUID="13432aa0-5907-4e63-8d12-cacad202b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 15:12:40.972557 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:40.972518    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"]
Nov 02 15:12:40.972996 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:40.972982    4891 cpu_manager.go:414] "containerMap: removing container" podUID="5726fda9-7b7b-4170-a62b-252d7f1d9ebd" containerName="revoke-keys"
Nov 02 15:12:40.973035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:40.972997    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5726fda9-7b7b-4170-a62b-252d7f1d9ebd" containerName="revoke-keys"
Nov 02 15:12:40.973144 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:40.973135    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="5726fda9-7b7b-4170-a62b-252d7f1d9ebd" containerName="revoke-keys"
Nov 02 15:12:40.977053 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:40.977020    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:41.000530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.000506    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsp2b\" (UniqueName: \"kubernetes.io/projected/4c01829a-ccdd-44d8-8187-fd36f8d25efa-kube-api-access-nsp2b\") pod \"maas-api-revoke-sub-0d50c9d3-k5n26\" (UID: \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:41.000686 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.000666    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d50c9d3-k5n26\" (UID: \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:41.000728 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.000700    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d50c9d3-k5n26\" (UID: \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:41.024168 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.024144    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"]
Nov 02 15:12:41.034004 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.033978    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:12:41.071023 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.069700    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:12:41.083649 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.083621    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:12:41.083832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.083663    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:12:41.101297 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.101270    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d50c9d3-k5n26\" (UID: \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:41.101297 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.101299    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d50c9d3-k5n26\" (UID: \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:41.101497 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.101347    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsp2b\" (UniqueName: \"kubernetes.io/projected/4c01829a-ccdd-44d8-8187-fd36f8d25efa-kube-api-access-nsp2b\") pod \"maas-api-revoke-sub-0d50c9d3-k5n26\" (UID: \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:41.102100 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.102075    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d50c9d3-k5n26\" (UID: \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:41.103605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.103586    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d50c9d3-k5n26\" (UID: \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:41.115658 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.115634    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qg548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:12:41.115738 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.115691    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" podUID="13432aa0-5907-4e63-8d12-cacad202b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 15:12:41.176591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.176558    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsp2b\" (UniqueName: \"kubernetes.io/projected/4c01829a-ccdd-44d8-8187-fd36f8d25efa-kube-api-access-nsp2b\") pod \"maas-api-revoke-sub-0d50c9d3-k5n26\" (UID: \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:41.192081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.192050    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"]
Nov 02 15:12:41.196159 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.196140    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:41.202508 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.202477    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-405f50c3-z29ft\" (UID: \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:41.202639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.202543    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28bp\" (UniqueName: \"kubernetes.io/projected/0e39bd5c-e83e-4c72-a399-ca15e5934a61-kube-api-access-m28bp\") pod \"maas-api-revoke-sub-405f50c3-z29ft\" (UID: \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:41.202707 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.202679    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-405f50c3-z29ft\" (UID: \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:41.273786 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.273659    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"]
Nov 02 15:12:41.290735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.290707    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:41.303750 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.303723    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-405f50c3-z29ft\" (UID: \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:41.303868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.303842    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-405f50c3-z29ft\" (UID: \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:41.303970 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.303932    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28bp\" (UniqueName: \"kubernetes.io/projected/0e39bd5c-e83e-4c72-a399-ca15e5934a61-kube-api-access-m28bp\") pod \"maas-api-revoke-sub-405f50c3-z29ft\" (UID: \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:41.304604 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.304582    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-405f50c3-z29ft\" (UID: \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:41.306130 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.306108    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-405f50c3-z29ft\" (UID: \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:41.381061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.377883    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28bp\" (UniqueName: \"kubernetes.io/projected/0e39bd5c-e83e-4c72-a399-ca15e5934a61-kube-api-access-m28bp\") pod \"maas-api-revoke-sub-405f50c3-z29ft\" (UID: \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:41.510496 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.510461    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:41.699028 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.699000    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"]
Nov 02 15:12:41.699158 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:41.699134    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c01829a_ccdd_44d8_8187_fd36f8d25efa.slice/crio-7c866735a196b357f778e7698afe2ead888d5fabfef5adce13c62cb40172b97c WatchSource:0}: Error finding container 7c866735a196b357f778e7698afe2ead888d5fabfef5adce13c62cb40172b97c: Status 404 returned error can't find the container with id 7c866735a196b357f778e7698afe2ead888d5fabfef5adce13c62cb40172b97c
Nov 02 15:12:41.915751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:41.915718    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"]
Nov 02 15:12:41.916595 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:41.916568    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e39bd5c_e83e_4c72_a399_ca15e5934a61.slice/crio-563b9d24130cfcb6785544bc4f9330b0153b0fd332a55b9e437c79a3f099fc25 WatchSource:0}: Error finding container 563b9d24130cfcb6785544bc4f9330b0153b0fd332a55b9e437c79a3f099fc25: Status 404 returned error can't find the container with id 563b9d24130cfcb6785544bc4f9330b0153b0fd332a55b9e437c79a3f099fc25
Nov 02 15:12:42.115743 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:42.115713    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qg548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:12:42.115892 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:42.115761    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" podUID="13432aa0-5907-4e63-8d12-cacad202b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 15:12:42.414772 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:42.414733    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft" event={"ID":"0e39bd5c-e83e-4c72-a399-ca15e5934a61","Type":"ContainerStarted","Data":"dfbf858181ce7174e4cb2a0eb213afdc02b5f4afbe83ed0e719d39ae008b6493"}
Nov 02 15:12:42.415392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:42.414801    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft" event={"ID":"0e39bd5c-e83e-4c72-a399-ca15e5934a61","Type":"ContainerStarted","Data":"563b9d24130cfcb6785544bc4f9330b0153b0fd332a55b9e437c79a3f099fc25"}
Nov 02 15:12:42.416361 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:42.416334    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26" event={"ID":"4c01829a-ccdd-44d8-8187-fd36f8d25efa","Type":"ContainerStarted","Data":"a5b33178f3b05e3e7b8098d47bb4a7e53253455488dadf8236c9777144518ed5"}
Nov 02 15:12:42.416509 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:42.416376    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26" event={"ID":"4c01829a-ccdd-44d8-8187-fd36f8d25efa","Type":"ContainerStarted","Data":"7c866735a196b357f778e7698afe2ead888d5fabfef5adce13c62cb40172b97c"}
Nov 02 15:12:42.535944 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:42.535889    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft" podStartSLOduration=1.53587582 podStartE2EDuration="1.53587582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:42.532909443 +0000 UTC m=+970.125526280" watchObservedRunningTime="2026-11-02 15:12:42.53587582 +0000 UTC m=+970.128492620"
Nov 02 15:12:42.675707 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:42.673400    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26" podStartSLOduration=2.673380482 podStartE2EDuration="2.673380482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:42.661071923 +0000 UTC m=+970.253688724" watchObservedRunningTime="2026-11-02 15:12:42.673380482 +0000 UTC m=+970.265997282"
Nov 02 15:12:43.116015 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:43.115980    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qg548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:12:43.116188 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:43.116036    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" podUID="13432aa0-5907-4e63-8d12-cacad202b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 15:12:43.424046 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:43.423958    4891 generic.go:356] "Generic (PLEG): container finished" podID="0e39bd5c-e83e-4c72-a399-ca15e5934a61" containerID="dfbf858181ce7174e4cb2a0eb213afdc02b5f4afbe83ed0e719d39ae008b6493" exitCode=0
Nov 02 15:12:43.424504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:43.424049    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft" event={"ID":"0e39bd5c-e83e-4c72-a399-ca15e5934a61","Type":"ContainerDied","Data":"dfbf858181ce7174e4cb2a0eb213afdc02b5f4afbe83ed0e719d39ae008b6493"}
Nov 02 15:12:43.430358 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:43.430329    4891 generic.go:356] "Generic (PLEG): container finished" podID="4c01829a-ccdd-44d8-8187-fd36f8d25efa" containerID="a5b33178f3b05e3e7b8098d47bb4a7e53253455488dadf8236c9777144518ed5" exitCode=0
Nov 02 15:12:43.430493 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:43.430417    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26" event={"ID":"4c01829a-ccdd-44d8-8187-fd36f8d25efa","Type":"ContainerDied","Data":"a5b33178f3b05e3e7b8098d47bb4a7e53253455488dadf8236c9777144518ed5"}
Nov 02 15:12:44.116014 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.115981    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qg548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:12:44.116189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.116034    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" podUID="13432aa0-5907-4e63-8d12-cacad202b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 15:12:44.273452 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:12:44.273417    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:12:44.596211 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.596180    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:44.634565 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.634540    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-openshift-ca\") pod \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\" (UID: \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\") "
Nov 02 15:12:44.634714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.634624    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e39bd5c-e83e-4c72-a399-ca15e5934a61-kube-api-access-m28bp\" (UniqueName: \"kubernetes.io/projected/0e39bd5c-e83e-4c72-a399-ca15e5934a61-kube-api-access-m28bp\") pod \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\" (UID: \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\") "
Nov 02 15:12:44.634714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.634653    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-xks-ca\") pod \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\" (UID: \"0e39bd5c-e83e-4c72-a399-ca15e5934a61\") "
Nov 02 15:12:44.634988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.634960    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-openshift-ca" pod "0e39bd5c-e83e-4c72-a399-ca15e5934a61" (UID: "0e39bd5c-e83e-4c72-a399-ca15e5934a61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:44.636791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.636763    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-xks-ca" pod "0e39bd5c-e83e-4c72-a399-ca15e5934a61" (UID: "0e39bd5c-e83e-4c72-a399-ca15e5934a61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:44.636877 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.636804    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e39bd5c-e83e-4c72-a399-ca15e5934a61-kube-api-access-m28bp" pod "0e39bd5c-e83e-4c72-a399-ca15e5934a61" (UID: "0e39bd5c-e83e-4c72-a399-ca15e5934a61"). InnerVolumeSpecName "kube-api-access-m28bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:44.659849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.659825    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:44.735972 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.735898    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-xks-ca\") pod \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\" (UID: \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\") "
Nov 02 15:12:44.735972 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.735944    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c01829a-ccdd-44d8-8187-fd36f8d25efa-kube-api-access-nsp2b\" (UniqueName: \"kubernetes.io/projected/4c01829a-ccdd-44d8-8187-fd36f8d25efa-kube-api-access-nsp2b\") pod \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\" (UID: \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\") "
Nov 02 15:12:44.736156 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.736069    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-openshift-ca\") pod \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\" (UID: \"4c01829a-ccdd-44d8-8187-fd36f8d25efa\") "
Nov 02 15:12:44.736502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.736479    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:44.736502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.736490    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-openshift-ca" pod "4c01829a-ccdd-44d8-8187-fd36f8d25efa" (UID: "4c01829a-ccdd-44d8-8187-fd36f8d25efa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:44.736502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.736503    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m28bp\" (UniqueName: \"kubernetes.io/projected/0e39bd5c-e83e-4c72-a399-ca15e5934a61-kube-api-access-m28bp\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:44.736750 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.736519    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e39bd5c-e83e-4c72-a399-ca15e5934a61-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:44.738162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.738141    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-xks-ca" pod "4c01829a-ccdd-44d8-8187-fd36f8d25efa" (UID: "4c01829a-ccdd-44d8-8187-fd36f8d25efa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:44.738268 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.738185    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c01829a-ccdd-44d8-8187-fd36f8d25efa-kube-api-access-nsp2b" pod "4c01829a-ccdd-44d8-8187-fd36f8d25efa" (UID: "4c01829a-ccdd-44d8-8187-fd36f8d25efa"). InnerVolumeSpecName "kube-api-access-nsp2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:44.837887 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.837562    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:44.837887 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.837592    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsp2b\" (UniqueName: \"kubernetes.io/projected/4c01829a-ccdd-44d8-8187-fd36f8d25efa-kube-api-access-nsp2b\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:44.837887 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:44.837603    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c01829a-ccdd-44d8-8187-fd36f8d25efa-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:45.116337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:45.116279    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qg548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:12:45.116520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:45.116377    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" podUID="13432aa0-5907-4e63-8d12-cacad202b86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 15:12:45.451214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:45.451123    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"
Nov 02 15:12:45.451214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:45.451136    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft" event={"ID":"0e39bd5c-e83e-4c72-a399-ca15e5934a61","Type":"ContainerDied","Data":"563b9d24130cfcb6785544bc4f9330b0153b0fd332a55b9e437c79a3f099fc25"}
Nov 02 15:12:45.451214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:45.451191    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="563b9d24130cfcb6785544bc4f9330b0153b0fd332a55b9e437c79a3f099fc25"
Nov 02 15:12:45.452788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:45.452751    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26" event={"ID":"4c01829a-ccdd-44d8-8187-fd36f8d25efa","Type":"ContainerDied","Data":"7c866735a196b357f778e7698afe2ead888d5fabfef5adce13c62cb40172b97c"}
Nov 02 15:12:45.452900 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:45.452804    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c866735a196b357f778e7698afe2ead888d5fabfef5adce13c62cb40172b97c"
Nov 02 15:12:45.452900 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:45.452765    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"
Nov 02 15:12:46.120474 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:46.120445    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:46.120922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:46.120904    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:46.121682 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:46.121665    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:12:47.929018 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.928979    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"]
Nov 02 15:12:47.929729 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.929706    4891 cpu_manager.go:414] "containerMap: removing container" podUID="0e39bd5c-e83e-4c72-a399-ca15e5934a61" containerName="revoke-keys"
Nov 02 15:12:47.929781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.929732    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e39bd5c-e83e-4c72-a399-ca15e5934a61" containerName="revoke-keys"
Nov 02 15:12:47.929781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.929757    4891 cpu_manager.go:414] "containerMap: removing container" podUID="4c01829a-ccdd-44d8-8187-fd36f8d25efa" containerName="revoke-keys"
Nov 02 15:12:47.929781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.929766    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c01829a-ccdd-44d8-8187-fd36f8d25efa" containerName="revoke-keys"
Nov 02 15:12:47.929926 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.929913    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c01829a-ccdd-44d8-8187-fd36f8d25efa" containerName="revoke-keys"
Nov 02 15:12:47.929961 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.929941    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e39bd5c-e83e-4c72-a399-ca15e5934a61" containerName="revoke-keys"
Nov 02 15:12:47.933896 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.933877    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:47.984556 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.984511    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:47.984799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.984625    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:47.984799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.984653    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:47.984799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.984707    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:47.984799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.984767    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slvmx\" (UniqueName: \"kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-kube-api-access-slvmx\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:47.985036 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.984828    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:47.985036 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.984863    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:47.985036 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.984897    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:47.985036 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.984936    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014837c5-3da0-4c7d-a46a-92a84dac45fc-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:47.987819 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:47.987791    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-de35cd-openshift-default-dockercfg-wdrm7\""
Nov 02 15:12:48.042870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.042830    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"]
Nov 02 15:12:48.086507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.086478    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.086692 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.086572    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.086692 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.086602    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.086692 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.086655    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.086859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.086729    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slvmx\" (UniqueName: \"kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-kube-api-access-slvmx\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.086859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.086792    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.086859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.086826    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.087013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.086868    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.087013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.086909    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014837c5-3da0-4c7d-a46a-92a84dac45fc-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.087120 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.087050    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.087120 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.087108    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.087240 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.087157    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.087240 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.087204    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.087651 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.087627    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014837c5-3da0-4c7d-a46a-92a84dac45fc-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.088881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.088863    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.089279 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.089262    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.161331 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.151798    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slvmx\" (UniqueName: \"kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-kube-api-access-slvmx\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.161331 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.159834    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.247919 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.247829    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:48.691530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.691503    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"]
Nov 02 15:12:48.692520 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:48.692491    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014837c5_3da0_4c7d_a46a_92a84dac45fc.slice/crio-f66c1657e84d0a5ebe7f45253a3b3acc9b18d05075833d7e48d14ec4a8c089a0 WatchSource:0}: Error finding container f66c1657e84d0a5ebe7f45253a3b3acc9b18d05075833d7e48d14ec4a8c089a0: Status 404 returned error can't find the container with id f66c1657e84d0a5ebe7f45253a3b3acc9b18d05075833d7e48d14ec4a8c089a0
Nov 02 15:12:48.695641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.695600    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:48.695734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.695692    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:48.695774 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:48.695733    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:49.484491 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:49.484453    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7" event={"ID":"014837c5-3da0-4c7d-a46a-92a84dac45fc","Type":"ContainerStarted","Data":"515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2"}
Nov 02 15:12:49.484952 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:49.484506    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7" event={"ID":"014837c5-3da0-4c7d-a46a-92a84dac45fc","Type":"ContainerStarted","Data":"f66c1657e84d0a5ebe7f45253a3b3acc9b18d05075833d7e48d14ec4a8c089a0"}
Nov 02 15:12:49.611329 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:49.611270    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7" podStartSLOduration=2.611256893 podStartE2EDuration="2.611256893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:49.608096921 +0000 UTC m=+977.200713793" watchObservedRunningTime="2026-11-02 15:12:49.611256893 +0000 UTC m=+977.203873705"
Nov 02 15:12:50.249335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:50.248644    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:50.259334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:50.258454    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 15:12:50.259334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:50.258537    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7" podUID="014837c5-3da0-4c7d-a46a-92a84dac45fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 15:12:51.253741 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.253701    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 15:12:51.254346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.254291    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7" podUID="014837c5-3da0-4c7d-a46a-92a84dac45fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 15:12:51.793256 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.793218    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"]
Nov 02 15:12:51.798089 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.798068    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:51.829570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.829537    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-926qd\" (UniqueName: \"kubernetes.io/projected/191aab13-55f5-41fa-8c4f-c615ad7f21f6-kube-api-access-926qd\") pod \"maas-api-revoke-sub-138d9353-x6fwj\" (UID: \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:51.829748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.829586    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-138d9353-x6fwj\" (UID: \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:51.829802 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.829781    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-138d9353-x6fwj\" (UID: \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:51.876059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.876023    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"]
Nov 02 15:12:51.916677 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.916646    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:12:51.916677 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.916669    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:12:51.916677 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.916676    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:12:51.916923 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.916738    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:12:51.930703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.930679    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-138d9353-x6fwj\" (UID: \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:51.930854 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.930825    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-138d9353-x6fwj\" (UID: \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:51.930854 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.930847    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-926qd\" (UniqueName: \"kubernetes.io/projected/191aab13-55f5-41fa-8c4f-c615ad7f21f6-kube-api-access-926qd\") pod \"maas-api-revoke-sub-138d9353-x6fwj\" (UID: \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:51.931611 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.931583    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-138d9353-x6fwj\" (UID: \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:51.932933 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.932906    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-138d9353-x6fwj\" (UID: \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:51.995396 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:51.995368    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-926qd\" (UniqueName: \"kubernetes.io/projected/191aab13-55f5-41fa-8c4f-c615ad7f21f6-kube-api-access-926qd\") pod \"maas-api-revoke-sub-138d9353-x6fwj\" (UID: \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:52.113456 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:52.113367    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:52.269336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:52.256725    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 15:12:52.269336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:52.256787    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7" podUID="014837c5-3da0-4c7d-a46a-92a84dac45fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 15:12:52.527074 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:52.527042    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"]
Nov 02 15:12:52.528567 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:12:52.528537    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191aab13_55f5_41fa_8c4f_c615ad7f21f6.slice/crio-5f48ee644d131ea27128f0aa90fdb570a74325e8948f747bd5f7c7d84f09cb61 WatchSource:0}: Error finding container 5f48ee644d131ea27128f0aa90fdb570a74325e8948f747bd5f7c7d84f09cb61: Status 404 returned error can't find the container with id 5f48ee644d131ea27128f0aa90fdb570a74325e8948f747bd5f7c7d84f09cb61
Nov 02 15:12:53.263797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:53.263708    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 15:12:53.264054 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:53.264027    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7" podUID="014837c5-3da0-4c7d-a46a-92a84dac45fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 15:12:53.513933 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:53.513834    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj" event={"ID":"191aab13-55f5-41fa-8c4f-c615ad7f21f6","Type":"ContainerStarted","Data":"cdf5f6b4eaaabaf7426dc5fd591d5f7a96fa2510d46e7b6584976ab86186ffc0"}
Nov 02 15:12:53.513933 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:53.513920    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj" event={"ID":"191aab13-55f5-41fa-8c4f-c615ad7f21f6","Type":"ContainerStarted","Data":"5f48ee644d131ea27128f0aa90fdb570a74325e8948f747bd5f7c7d84f09cb61"}
Nov 02 15:12:53.637290 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:53.637231    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj" podStartSLOduration=2.63721547 podStartE2EDuration="2.63721547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:53.6329529 +0000 UTC m=+981.225569706" watchObservedRunningTime="2026-11-02 15:12:53.63721547 +0000 UTC m=+981.229832268"
Nov 02 15:12:54.263335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:54.259735    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 15:12:54.263335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:54.259804    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7" podUID="014837c5-3da0-4c7d-a46a-92a84dac45fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 15:12:54.524782 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:54.524751    4891 generic.go:356] "Generic (PLEG): container finished" podID="191aab13-55f5-41fa-8c4f-c615ad7f21f6" containerID="cdf5f6b4eaaabaf7426dc5fd591d5f7a96fa2510d46e7b6584976ab86186ffc0" exitCode=0
Nov 02 15:12:54.525197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:54.524839    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj" event={"ID":"191aab13-55f5-41fa-8c4f-c615ad7f21f6","Type":"ContainerDied","Data":"cdf5f6b4eaaabaf7426dc5fd591d5f7a96fa2510d46e7b6584976ab86186ffc0"}
Nov 02 15:12:55.481900 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.481861    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:55.482424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.482395    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:55.484025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.484003    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:12:55.686116 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.686089    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:55.801588 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.801551    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-xks-ca\") pod \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\" (UID: \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\") "
Nov 02 15:12:55.801806 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.801645    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191aab13-55f5-41fa-8c4f-c615ad7f21f6-kube-api-access-926qd\" (UniqueName: \"kubernetes.io/projected/191aab13-55f5-41fa-8c4f-c615ad7f21f6-kube-api-access-926qd\") pod \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\" (UID: \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\") "
Nov 02 15:12:55.801891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.801817    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-openshift-ca\") pod \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\" (UID: \"191aab13-55f5-41fa-8c4f-c615ad7f21f6\") "
Nov 02 15:12:55.802458 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.802428    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-openshift-ca" pod "191aab13-55f5-41fa-8c4f-c615ad7f21f6" (UID: "191aab13-55f5-41fa-8c4f-c615ad7f21f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:12:55.804601 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.804562    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191aab13-55f5-41fa-8c4f-c615ad7f21f6-kube-api-access-926qd" pod "191aab13-55f5-41fa-8c4f-c615ad7f21f6" (UID: "191aab13-55f5-41fa-8c4f-c615ad7f21f6"). InnerVolumeSpecName "kube-api-access-926qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:12:55.804830 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.804798    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-xks-ca" pod "191aab13-55f5-41fa-8c4f-c615ad7f21f6" (UID: "191aab13-55f5-41fa-8c4f-c615ad7f21f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:12:55.903635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.903600    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:55.903789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.903638    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191aab13-55f5-41fa-8c4f-c615ad7f21f6-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:55.903789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:55.903656    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-926qd\" (UniqueName: \"kubernetes.io/projected/191aab13-55f5-41fa-8c4f-c615ad7f21f6-kube-api-access-926qd\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:12:56.540373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:56.540343    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"
Nov 02 15:12:56.540537 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:56.540342    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj" event={"ID":"191aab13-55f5-41fa-8c4f-c615ad7f21f6","Type":"ContainerDied","Data":"5f48ee644d131ea27128f0aa90fdb570a74325e8948f747bd5f7c7d84f09cb61"}
Nov 02 15:12:56.540537 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:12:56.540464    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f48ee644d131ea27128f0aa90fdb570a74325e8948f747bd5f7c7d84f09cb61"
Nov 02 15:13:03.341534 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.341497    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"]
Nov 02 15:13:03.342011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.341996    4891 cpu_manager.go:414] "containerMap: removing container" podUID="191aab13-55f5-41fa-8c4f-c615ad7f21f6" containerName="revoke-keys"
Nov 02 15:13:03.342055 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.342011    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191aab13-55f5-41fa-8c4f-c615ad7f21f6" containerName="revoke-keys"
Nov 02 15:13:03.342101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.342089    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="191aab13-55f5-41fa-8c4f-c615ad7f21f6" containerName="revoke-keys"
Nov 02 15:13:03.345644 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.345628    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:03.398844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.398810    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:13:03.408499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.408463    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8c9a309-8m576\" (UID: \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:03.408668 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.408626    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8c9a309-8m576\" (UID: \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:03.408724 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.408709    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7t9c\" (UniqueName: \"kubernetes.io/projected/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-kube-api-access-q7t9c\") pod \"maas-api-revoke-sub-b8c9a309-8m576\" (UID: \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:03.448473 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.448437    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:13:03.448473 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.448471    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:13:03.448711 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.448470    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:13:03.452887 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.452865    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"]
Nov 02 15:13:03.509943 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.509903    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7t9c\" (UniqueName: \"kubernetes.io/projected/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-kube-api-access-q7t9c\") pod \"maas-api-revoke-sub-b8c9a309-8m576\" (UID: \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:03.510128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.510050    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8c9a309-8m576\" (UID: \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:03.510187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.510141    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8c9a309-8m576\" (UID: \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:03.511019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.510990    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8c9a309-8m576\" (UID: \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:03.512781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.512764    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8c9a309-8m576\" (UID: \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:03.570881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.570848    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7t9c\" (UniqueName: \"kubernetes.io/projected/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-kube-api-access-q7t9c\") pod \"maas-api-revoke-sub-b8c9a309-8m576\" (UID: \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:03.659206 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:03.659129    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:04.039259 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:04.039228    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"]
Nov 02 15:13:04.039916 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:13:04.039889    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddadd3c9_7301_4ffb_ac4c_8bdc629bccf8.slice/crio-b2ab74e4c7e6d2dfa91c0cc945f9fda7e224b68cda597935604c71b4bcab0e00 WatchSource:0}: Error finding container b2ab74e4c7e6d2dfa91c0cc945f9fda7e224b68cda597935604c71b4bcab0e00: Status 404 returned error can't find the container with id b2ab74e4c7e6d2dfa91c0cc945f9fda7e224b68cda597935604c71b4bcab0e00
Nov 02 15:13:04.596241 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:04.596209    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576" event={"ID":"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8","Type":"ContainerStarted","Data":"f6456c4c14c47d73ddf2044e47858d74d0c42e598e6dbb04196e7ef5be6fc6cc"}
Nov 02 15:13:04.596620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:04.596257    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576" event={"ID":"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8","Type":"ContainerStarted","Data":"b2ab74e4c7e6d2dfa91c0cc945f9fda7e224b68cda597935604c71b4bcab0e00"}
Nov 02 15:13:04.763976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:04.763875    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576" podStartSLOduration=1.763860585 podStartE2EDuration="1.763860585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:04.71201094 +0000 UTC m=+992.304627740" watchObservedRunningTime="2026-11-02 15:13:04.763860585 +0000 UTC m=+992.356477383"
Nov 02 15:13:05.604126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:05.604084    4891 generic.go:356] "Generic (PLEG): container finished" podID="ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8" containerID="f6456c4c14c47d73ddf2044e47858d74d0c42e598e6dbb04196e7ef5be6fc6cc" exitCode=0
Nov 02 15:13:05.604551 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:05.604167    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576" event={"ID":"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8","Type":"ContainerDied","Data":"f6456c4c14c47d73ddf2044e47858d74d0c42e598e6dbb04196e7ef5be6fc6cc"}
Nov 02 15:13:06.752696 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:06.752675    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:06.846829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:06.846795    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-kube-api-access-q7t9c\" (UniqueName: \"kubernetes.io/projected/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-kube-api-access-q7t9c\") pod \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\" (UID: \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\") "
Nov 02 15:13:06.846998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:06.846895    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-xks-ca\") pod \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\" (UID: \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\") "
Nov 02 15:13:06.846998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:06.846991    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-openshift-ca\") pod \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\" (UID: \"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8\") "
Nov 02 15:13:06.847383 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:06.847355    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-openshift-ca" pod "ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8" (UID: "ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:06.847496 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:06.847441    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:13:06.848954 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:06.848930    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-xks-ca" pod "ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8" (UID: "ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:06.849046 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:06.848982    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-kube-api-access-q7t9c" pod "ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8" (UID: "ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8"). InnerVolumeSpecName "kube-api-access-q7t9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:06.948064 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:06.947985    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7t9c\" (UniqueName: \"kubernetes.io/projected/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-kube-api-access-q7t9c\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:13:06.948064 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:06.948012    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:13:07.619012 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:07.618980    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"
Nov 02 15:13:07.619012 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:07.618996    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576" event={"ID":"ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8","Type":"ContainerDied","Data":"b2ab74e4c7e6d2dfa91c0cc945f9fda7e224b68cda597935604c71b4bcab0e00"}
Nov 02 15:13:07.619224 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:07.619038    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2ab74e4c7e6d2dfa91c0cc945f9fda7e224b68cda597935604c71b4bcab0e00"
Nov 02 15:13:14.274051 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:13:14.274010    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:13:28.838611 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.838579    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"]
Nov 02 15:13:28.839051 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.839038    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8" containerName="revoke-keys"
Nov 02 15:13:28.839104 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.839052    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8" containerName="revoke-keys"
Nov 02 15:13:28.839141 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.839127    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8" containerName="revoke-keys"
Nov 02 15:13:28.842481 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.842464    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:28.917909 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.917876    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"]
Nov 02 15:13:28.962004 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.961972    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:13:28.962267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.962247    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:13:28.962354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.962247    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:13:28.962354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.962285    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:13:28.986868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.986831    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d38bc66-4lfbr\" (UID: \"96e1a797-bf38-4019-a144-1ee24dd8ce60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:28.987034 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.986963    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xht9\" (UniqueName: \"kubernetes.io/projected/96e1a797-bf38-4019-a144-1ee24dd8ce60-kube-api-access-8xht9\") pod \"maas-api-revoke-sub-6d38bc66-4lfbr\" (UID: \"96e1a797-bf38-4019-a144-1ee24dd8ce60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:28.987093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:28.987045    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d38bc66-4lfbr\" (UID: \"96e1a797-bf38-4019-a144-1ee24dd8ce60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:29.088402 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:29.088364    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d38bc66-4lfbr\" (UID: \"96e1a797-bf38-4019-a144-1ee24dd8ce60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:29.088603 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:29.088506    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d38bc66-4lfbr\" (UID: \"96e1a797-bf38-4019-a144-1ee24dd8ce60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:29.088603 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:29.088587    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xht9\" (UniqueName: \"kubernetes.io/projected/96e1a797-bf38-4019-a144-1ee24dd8ce60-kube-api-access-8xht9\") pod \"maas-api-revoke-sub-6d38bc66-4lfbr\" (UID: \"96e1a797-bf38-4019-a144-1ee24dd8ce60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:29.089223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:29.089169    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d38bc66-4lfbr\" (UID: \"96e1a797-bf38-4019-a144-1ee24dd8ce60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:29.090875 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:29.090853    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d38bc66-4lfbr\" (UID: \"96e1a797-bf38-4019-a144-1ee24dd8ce60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:29.150398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:29.150370    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xht9\" (UniqueName: \"kubernetes.io/projected/96e1a797-bf38-4019-a144-1ee24dd8ce60-kube-api-access-8xht9\") pod \"maas-api-revoke-sub-6d38bc66-4lfbr\" (UID: \"96e1a797-bf38-4019-a144-1ee24dd8ce60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:29.155187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:29.155157    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:29.543114 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:29.543090    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"]
Nov 02 15:13:29.545875 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:13:29.545845    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e1a797_bf38_4019_a144_1ee24dd8ce60.slice/crio-776f9f1e83e1a1895af456b247978434478eb8f2a601b62b22595d99411d2c80 WatchSource:0}: Error finding container 776f9f1e83e1a1895af456b247978434478eb8f2a601b62b22595d99411d2c80: Status 404 returned error can't find the container with id 776f9f1e83e1a1895af456b247978434478eb8f2a601b62b22595d99411d2c80
Nov 02 15:13:29.803612 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:29.803506    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr" event={"ID":"96e1a797-bf38-4019-a144-1ee24dd8ce60","Type":"ContainerStarted","Data":"776f9f1e83e1a1895af456b247978434478eb8f2a601b62b22595d99411d2c80"}
Nov 02 15:13:30.813909 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:30.813819    4891 generic.go:356] "Generic (PLEG): container finished" podID="96e1a797-bf38-4019-a144-1ee24dd8ce60" containerID="3f580c12c5917a63c594ae7f2530db8d9eac57d30f7208d4a57ed8d8bdef6676" exitCode=0
Nov 02 15:13:30.814273 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:30.813904    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr" event={"ID":"96e1a797-bf38-4019-a144-1ee24dd8ce60","Type":"ContainerDied","Data":"3f580c12c5917a63c594ae7f2530db8d9eac57d30f7208d4a57ed8d8bdef6676"}
Nov 02 15:13:31.981859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:31.981830    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:32.123556 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:32.123472    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e1a797-bf38-4019-a144-1ee24dd8ce60-kube-api-access-8xht9\" (UniqueName: \"kubernetes.io/projected/96e1a797-bf38-4019-a144-1ee24dd8ce60-kube-api-access-8xht9\") pod \"96e1a797-bf38-4019-a144-1ee24dd8ce60\" (UID: \"96e1a797-bf38-4019-a144-1ee24dd8ce60\") "
Nov 02 15:13:32.123713 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:32.123577    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-openshift-ca\") pod \"96e1a797-bf38-4019-a144-1ee24dd8ce60\" (UID: \"96e1a797-bf38-4019-a144-1ee24dd8ce60\") "
Nov 02 15:13:32.123713 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:32.123708    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-xks-ca\") pod \"96e1a797-bf38-4019-a144-1ee24dd8ce60\" (UID: \"96e1a797-bf38-4019-a144-1ee24dd8ce60\") "
Nov 02 15:13:32.124017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:32.123986    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-openshift-ca" pod "96e1a797-bf38-4019-a144-1ee24dd8ce60" (UID: "96e1a797-bf38-4019-a144-1ee24dd8ce60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:32.125801 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:32.125765    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e1a797-bf38-4019-a144-1ee24dd8ce60-kube-api-access-8xht9" pod "96e1a797-bf38-4019-a144-1ee24dd8ce60" (UID: "96e1a797-bf38-4019-a144-1ee24dd8ce60"). InnerVolumeSpecName "kube-api-access-8xht9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:32.125922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:32.125806    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-xks-ca" pod "96e1a797-bf38-4019-a144-1ee24dd8ce60" (UID: "96e1a797-bf38-4019-a144-1ee24dd8ce60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:32.224654 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:32.224620    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:13:32.224654 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:32.224652    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e1a797-bf38-4019-a144-1ee24dd8ce60-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:13:32.224654 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:32.224662    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xht9\" (UniqueName: \"kubernetes.io/projected/96e1a797-bf38-4019-a144-1ee24dd8ce60-kube-api-access-8xht9\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:13:32.830915 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:32.830839    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"
Nov 02 15:13:32.830915 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:32.830871    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr" event={"ID":"96e1a797-bf38-4019-a144-1ee24dd8ce60","Type":"ContainerDied","Data":"776f9f1e83e1a1895af456b247978434478eb8f2a601b62b22595d99411d2c80"}
Nov 02 15:13:32.831095 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:32.830917    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="776f9f1e83e1a1895af456b247978434478eb8f2a601b62b22595d99411d2c80"
Nov 02 15:13:34.036220 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.036181    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"]
Nov 02 15:13:34.036925 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.036903    4891 cpu_manager.go:414] "containerMap: removing container" podUID="96e1a797-bf38-4019-a144-1ee24dd8ce60" containerName="revoke-keys"
Nov 02 15:13:34.037017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.036927    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e1a797-bf38-4019-a144-1ee24dd8ce60" containerName="revoke-keys"
Nov 02 15:13:34.037074 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.037065    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e1a797-bf38-4019-a144-1ee24dd8ce60" containerName="revoke-keys"
Nov 02 15:13:34.040537 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.040516    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.143419 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.143387    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c18e79\"/\"distinct-w6-c18e79-kserve-self-signed-certs\""
Nov 02 15:13:34.145942 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.145911    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc69k\" (UniqueName: \"kubernetes.io/projected/75845278-3169-49ce-ace9-40a8a6ee001a-kube-api-access-bc69k\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.146090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.146014    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-home\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.146090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.146052    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-tmp-dir\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.146090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.146077    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-model-cache\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.146262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.146145    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75845278-3169-49ce-ace9-40a8a6ee001a-tls-certs\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.146262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.146178    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-dshm\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.147508 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.147484    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"]
Nov 02 15:13:34.247026 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.246993    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc69k\" (UniqueName: \"kubernetes.io/projected/75845278-3169-49ce-ace9-40a8a6ee001a-kube-api-access-bc69k\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.247270 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.247079    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-home\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.247270 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.247117    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-tmp-dir\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.247270 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.247147    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-model-cache\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.247270 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.247183    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75845278-3169-49ce-ace9-40a8a6ee001a-tls-certs\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.247270 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.247225    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-dshm\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.247621 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.247594    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-home\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.247678 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.247641    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-model-cache\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.248061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.248038    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-tmp-dir\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.250047 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.250021    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-dshm\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.250301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.250282    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75845278-3169-49ce-ace9-40a8a6ee001a-tls-certs\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.308037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.307966    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc69k\" (UniqueName: \"kubernetes.io/projected/75845278-3169-49ce-ace9-40a8a6ee001a-kube-api-access-bc69k\") pod \"distinct-w6-c18e79-kserve-5546557dcb-6hg5b\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") " pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.352555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.352517    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:34.778287 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:13:34.778257    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75845278_3169_49ce_ace9_40a8a6ee001a.slice/crio-ec0d85ea06d0e562bde6dd5645653ca8cbb09c612c4b312a0b4dd3c392358d55 WatchSource:0}: Error finding container ec0d85ea06d0e562bde6dd5645653ca8cbb09c612c4b312a0b4dd3c392358d55: Status 404 returned error can't find the container with id ec0d85ea06d0e562bde6dd5645653ca8cbb09c612c4b312a0b4dd3c392358d55
Nov 02 15:13:34.778650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.778612    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"]
Nov 02 15:13:34.848904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.848873    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" event={"ID":"75845278-3169-49ce-ace9-40a8a6ee001a","Type":"ContainerStarted","Data":"8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968"}
Nov 02 15:13:34.849059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.848919    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" event={"ID":"75845278-3169-49ce-ace9-40a8a6ee001a","Type":"ContainerStarted","Data":"ec0d85ea06d0e562bde6dd5645653ca8cbb09c612c4b312a0b4dd3c392358d55"}
Nov 02 15:13:34.973029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:34.972978    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podStartSLOduration=1.972963791 podStartE2EDuration="1.972963791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:34.968969153 +0000 UTC m=+1022.561585952" watchObservedRunningTime="2026-11-02 15:13:34.972963791 +0000 UTC m=+1022.565580592"
Nov 02 15:13:35.352966 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:35.352911    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:37.323024 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.322992    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"]
Nov 02 15:13:37.327024 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.327002    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.385282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.385244    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-db3931\"/\"distinct-w5-db3931-kserve-self-signed-certs\""
Nov 02 15:13:37.438780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.438745    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"]
Nov 02 15:13:37.486910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.486872    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-dshm\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.487086 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.486919    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66fd185-3f21-4783-b080-72c2283e0152-tls-certs\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.487086 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.487024    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-model-cache\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.487157 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.487082    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-home\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.487222 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.487201    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-tmp-dir\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.487264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.487250    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs6nl\" (UniqueName: \"kubernetes.io/projected/f66fd185-3f21-4783-b080-72c2283e0152-kube-api-access-qs6nl\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.588330 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.588236    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs6nl\" (UniqueName: \"kubernetes.io/projected/f66fd185-3f21-4783-b080-72c2283e0152-kube-api-access-qs6nl\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.588330 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.588301    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-dshm\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.588546 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.588349    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66fd185-3f21-4783-b080-72c2283e0152-tls-certs\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.588546 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.588529    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-model-cache\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.588660 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.588624    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-home\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.588755 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.588736    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-tmp-dir\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.588895 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.588867    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-model-cache\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.589150 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.589130    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-tmp-dir\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.589268 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.589244    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-home\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.590703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.590677    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-dshm\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.591409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.591387    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66fd185-3f21-4783-b080-72c2283e0152-tls-certs\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.649539 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.649504    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs6nl\" (UniqueName: \"kubernetes.io/projected/f66fd185-3f21-4783-b080-72c2283e0152-kube-api-access-qs6nl\") pod \"distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") " pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:37.940096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:37.940016    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:38.325570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:38.325538    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"]
Nov 02 15:13:38.325909 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:13:38.325747    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf66fd185_3f21_4783_b080_72c2283e0152.slice/crio-a8ce802c37055524343225d57b2765ebde1b96459a6e0559475103603cba68ac WatchSource:0}: Error finding container a8ce802c37055524343225d57b2765ebde1b96459a6e0559475103603cba68ac: Status 404 returned error can't find the container with id a8ce802c37055524343225d57b2765ebde1b96459a6e0559475103603cba68ac
Nov 02 15:13:38.878233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:38.878195    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" event={"ID":"f66fd185-3f21-4783-b080-72c2283e0152","Type":"ContainerStarted","Data":"375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7"}
Nov 02 15:13:38.878435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:38.878243    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" event={"ID":"f66fd185-3f21-4783-b080-72c2283e0152","Type":"ContainerStarted","Data":"a8ce802c37055524343225d57b2765ebde1b96459a6e0559475103603cba68ac"}
Nov 02 15:13:38.940584 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:38.940540    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:38.997551 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:38.997488    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podStartSLOduration=1.997474107 podStartE2EDuration="1.997474107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:38.994818199 +0000 UTC m=+1026.587434999" watchObservedRunningTime="2026-11-02 15:13:38.997474107 +0000 UTC m=+1026.590090926"
Nov 02 15:13:44.274528 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:13:44.274455    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:13:44.352737 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:44.352708    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:44.365363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:44.365334    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:44.937733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:44.937702    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:13:47.941024 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:47.940990    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:47.954859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:47.954831    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:13:48.956423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:13:48.956393    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:14:12.146971 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.146934    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"]
Nov 02 15:14:12.151155 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.151135    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.253858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.253823    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ba49d0\"/\"distinct-2-w0-ba49d0-kserve-self-signed-certs\""
Nov 02 15:14:12.258033 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.257997    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"]
Nov 02 15:14:12.290257 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.290224    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ca96d6-a378-43b9-855f-29b40a217715-tls-certs\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.290495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.290467    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-dshm\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.290583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.290515    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-model-cache\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.290631 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.290613    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p75bl\" (UniqueName: \"kubernetes.io/projected/59ca96d6-a378-43b9-855f-29b40a217715-kube-api-access-p75bl\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.290667 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.290653    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-home\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.290753 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.290738    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-tmp-dir\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.391748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.391704    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-dshm\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.391955 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.391762    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-model-cache\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.391955 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.391799    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p75bl\" (UniqueName: \"kubernetes.io/projected/59ca96d6-a378-43b9-855f-29b40a217715-kube-api-access-p75bl\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.391955 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.391827    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-home\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.391955 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.391890    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-tmp-dir\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.392201 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.391993    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ca96d6-a378-43b9-855f-29b40a217715-tls-certs\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.392263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.392217    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-model-cache\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.392375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.392343    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-home\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.392555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.392532    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-tmp-dir\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.394082 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.394059    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-dshm\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.394744 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.394721    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ca96d6-a378-43b9-855f-29b40a217715-tls-certs\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.453460 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.453384    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p75bl\" (UniqueName: \"kubernetes.io/projected/59ca96d6-a378-43b9-855f-29b40a217715-kube-api-access-p75bl\") pod \"distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") " pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.464103 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.464055    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:12.906084 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:12.905389    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"]
Nov 02 15:14:12.906084 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:14:12.905393    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59ca96d6_a378_43b9_855f_29b40a217715.slice/crio-5ed0bb23bc6e7ca9a71a928b4c44af4bc807abd57a62949cc85b7b0c0902e961 WatchSource:0}: Error finding container 5ed0bb23bc6e7ca9a71a928b4c44af4bc807abd57a62949cc85b7b0c0902e961: Status 404 returned error can't find the container with id 5ed0bb23bc6e7ca9a71a928b4c44af4bc807abd57a62949cc85b7b0c0902e961
Nov 02 15:14:13.144631 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:13.144593    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" event={"ID":"59ca96d6-a378-43b9-855f-29b40a217715","Type":"ContainerStarted","Data":"c7eefc33ae18808adad5b5c9c2f7963d85539e2d1208d2ad7a472118a02730a7"}
Nov 02 15:14:13.144797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:13.144645    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" event={"ID":"59ca96d6-a378-43b9-855f-29b40a217715","Type":"ContainerStarted","Data":"5ed0bb23bc6e7ca9a71a928b4c44af4bc807abd57a62949cc85b7b0c0902e961"}
Nov 02 15:14:13.265459 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:13.265361    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podStartSLOduration=1.265348833 podStartE2EDuration="1.265348833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:13.261446881 +0000 UTC m=+1060.854063679" watchObservedRunningTime="2026-11-02 15:14:13.265348833 +0000 UTC m=+1060.857965643"
Nov 02 15:14:13.465018 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:13.464983    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:14.276424 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:14:14.276384    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:14:20.360283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.360244    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"]
Nov 02 15:14:20.366279 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.366245    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.383782 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.383750    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.383931 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.383867    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.383931 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.383910    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.384046 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.384006    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b5sj\" (UniqueName: \"kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-kube-api-access-2b5sj\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.384108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.384094    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.384175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.384164    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.384213 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.384188    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.384213 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.384208    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.384303 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.384273    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.419509 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.419475    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-019efe-openshift-default-dockercfg-9lzm6\""
Nov 02 15:14:20.476355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.475840    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"]
Nov 02 15:14:20.485297 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.485241    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.486110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.485372    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.486110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.485417    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.486110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.485451    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.486110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.485503    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.486110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.485556    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.486110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.485621    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.486110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.485656    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.486110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.485712    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b5sj\" (UniqueName: \"kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-kube-api-access-2b5sj\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.486491 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.486448    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.486530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.486498    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.486788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.486767    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.486971 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.486949    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.487647 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.487620    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.489378 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.489349    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.495744 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.495725    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.547400 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.547371    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.547564 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.547460    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b5sj\" (UniqueName: \"kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-kube-api-access-2b5sj\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:20.684420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:20.684296    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:21.125468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:21.125440    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"]
Nov 02 15:14:21.125658 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:14:21.125633    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d189f67_43b9_4d7c_88fe_25e1bdb6c2b5.slice/crio-5c1b00a23bee43100c215037550048de77b4fa8694163894320f1c6f28ffc882 WatchSource:0}: Error finding container 5c1b00a23bee43100c215037550048de77b4fa8694163894320f1c6f28ffc882: Status 404 returned error can't find the container with id 5c1b00a23bee43100c215037550048de77b4fa8694163894320f1c6f28ffc882
Nov 02 15:14:21.127813 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:21.127780    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:14:21.127890 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:21.127845    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:14:21.127890 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:21.127878    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:14:21.203081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:21.203048    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" event={"ID":"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5","Type":"ContainerStarted","Data":"c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3"}
Nov 02 15:14:21.203225 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:21.203107    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" event={"ID":"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5","Type":"ContainerStarted","Data":"5c1b00a23bee43100c215037550048de77b4fa8694163894320f1c6f28ffc882"}
Nov 02 15:14:21.324336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:21.324258    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" podStartSLOduration=1.324245469 podStartE2EDuration="1.324245469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:21.323075834 +0000 UTC m=+1068.915692657" watchObservedRunningTime="2026-11-02 15:14:21.324245469 +0000 UTC m=+1068.916862268"
Nov 02 15:14:21.684431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:21.684395    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:22.471291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:22.471251    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:22.489735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:22.489706    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:22.695885 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:22.695406    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:14:22.695885 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:22.695465    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" podUID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 15:14:23.240206 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:23.240169    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:14:23.684948 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:23.684910    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:14:23.685130 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:23.684963    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" podUID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 15:14:24.685426 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:24.685340    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:14:24.685426 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:24.685406    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" podUID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 15:14:25.685637 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:25.685597    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:14:25.686138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:25.685652    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" podUID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 15:14:26.684681 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:26.684645    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:14:26.684874 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:26.684697    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" podUID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 15:14:27.685297 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:27.685251    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:14:27.685714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:27.685338    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" podUID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 15:14:28.689082 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:28.689050    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:28.689620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:28.689585    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:28.690242 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:28.690218    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:14:44.273762 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:14:44.273725    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:14:49.590482 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.590427    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"]
Nov 02 15:14:49.603619 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.603591    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.646806 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.646773    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.646957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.646816    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.646957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.646917    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.647032 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.646977    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bfe010-d72c-42f1-918c-8ec54254da96-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.647032 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.647021    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.647099 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.647089    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.647135 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.647124    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bfe010-d72c-42f1-918c-8ec54254da96-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.647175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.647163    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.647239 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.647226    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqt6c\" (UniqueName: \"kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-kube-api-access-kqt6c\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.655891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.655858    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ba49d0-openshift-default-dockercfg-66tsc\""
Nov 02 15:14:49.709583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.709548    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"]
Nov 02 15:14:49.748281 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.748244    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqt6c\" (UniqueName: \"kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-kube-api-access-kqt6c\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.748487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.748328    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.748487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.748362    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.748487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.748406    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.748487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.748434    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bfe010-d72c-42f1-918c-8ec54254da96-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.748737 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.748487    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.748737 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.748548    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.748737 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.748579    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bfe010-d72c-42f1-918c-8ec54254da96-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.748737 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.748647    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.748957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.748833    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.748957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.748862    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.749127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.749103    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.749214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.749193    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.749443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.749422    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bfe010-d72c-42f1-918c-8ec54254da96-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.750877 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.750847    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.751008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.750987    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bfe010-d72c-42f1-918c-8ec54254da96-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.810661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.810594    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.810834 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.810725    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqt6c\" (UniqueName: \"kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-kube-api-access-kqt6c\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:49.916771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:49.916693    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:50.349140 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:50.349111    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"]
Nov 02 15:14:50.349378 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:14:50.349352    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15bfe010_d72c_42f1_918c_8ec54254da96.slice/crio-05ed84586a91282985e76916c4d6950680f1d6183d97bd8b8041a9d99567850c WatchSource:0}: Error finding container 05ed84586a91282985e76916c4d6950680f1d6183d97bd8b8041a9d99567850c: Status 404 returned error can't find the container with id 05ed84586a91282985e76916c4d6950680f1d6183d97bd8b8041a9d99567850c
Nov 02 15:14:50.351647 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:50.351613    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:14:50.351740 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:50.351683    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:14:50.351740 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:50.351713    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:14:50.431900 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:50.431872    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" event={"ID":"15bfe010-d72c-42f1-918c-8ec54254da96","Type":"ContainerStarted","Data":"05ed84586a91282985e76916c4d6950680f1d6183d97bd8b8041a9d99567850c"}
Nov 02 15:14:50.559954 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:50.559892    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" podStartSLOduration=1.5598768280000002 podStartE2EDuration="1.559876828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:50.558989547 +0000 UTC m=+1098.151606345" watchObservedRunningTime="2026-11-02 15:14:50.559876828 +0000 UTC m=+1098.152493626"
Nov 02 15:14:50.917754 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:50.917721    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:51.440499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:51.440456    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" event={"ID":"15bfe010-d72c-42f1-918c-8ec54254da96","Type":"ContainerStarted","Data":"e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9"}
Nov 02 15:14:51.919344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:51.919278    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:14:51.919817 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:51.919348    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" podUID="15bfe010-d72c-42f1-918c-8ec54254da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 15:14:52.917485 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:52.917452    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:14:52.917693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:52.917519    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" podUID="15bfe010-d72c-42f1-918c-8ec54254da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 15:14:53.917510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:53.917471    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:14:53.917939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:53.917525    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" podUID="15bfe010-d72c-42f1-918c-8ec54254da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 15:14:54.917676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:54.917640    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:14:54.918080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:54.917690    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" podUID="15bfe010-d72c-42f1-918c-8ec54254da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 15:14:55.917728 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:55.917696    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:14:55.918126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:55.917750    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" podUID="15bfe010-d72c-42f1-918c-8ec54254da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 15:14:56.917354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:56.917285    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:14:56.917566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:56.917385    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" podUID="15bfe010-d72c-42f1-918c-8ec54254da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 15:14:57.922000 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:57.921966    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:57.922478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:57.922361    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:14:57.923009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:14:57.922981    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:15:00.191347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.191301    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"]
Nov 02 15:15:00.195620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.195597    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"]
Nov 02 15:15:00.195759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.195722    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"
Nov 02 15:15:00.200182 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.200163    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"]
Nov 02 15:15:00.200298 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.200280    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"
Nov 02 15:15:00.203912 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.203897    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"
Nov 02 15:15:00.240137 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.240110    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"]
Nov 02 15:15:00.243993 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.243971    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"
Nov 02 15:15:00.253670 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.253648    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:15:00.254024 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.254008    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:15:00.256819 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.256801    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"]
Nov 02 15:15:00.260566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.260550    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"
Nov 02 15:15:00.272103 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.272082    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwq7n\" (UniqueName: \"kubernetes.io/projected/305f5164-3c9f-452d-a175-63c432126262-kube-api-access-gwq7n\") pod \"maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb\" (UID: \"305f5164-3c9f-452d-a175-63c432126262\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"
Nov 02 15:15:00.272190 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.272145    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p\" (UID: \"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"
Nov 02 15:15:00.272247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.272228    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q9vs\" (UniqueName: \"kubernetes.io/projected/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-kube-api-access-2q9vs\") pod \"maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd\" (UID: \"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"
Nov 02 15:15:00.272288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.272276    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893875-jnj47\" (UID: \"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"
Nov 02 15:15:00.272416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.272398    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd\" (UID: \"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"
Nov 02 15:15:00.272457 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.272438    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mnrg\" (UniqueName: \"kubernetes.io/projected/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-kube-api-access-9mnrg\") pod \"maas-api-key-cleanup-29893875-jnj47\" (UID: \"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"
Nov 02 15:15:00.272491 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.272460    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305f5164-3c9f-452d-a175-63c432126262-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb\" (UID: \"305f5164-3c9f-452d-a175-63c432126262\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"
Nov 02 15:15:00.272564 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.272495    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znj2v\" (UniqueName: \"kubernetes.io/projected/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-kube-api-access-znj2v\") pod \"maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p\" (UID: \"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"
Nov 02 15:15:00.285606 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.285580    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:15:00.307045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.307011    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"]
Nov 02 15:15:00.326604 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.326552    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"]
Nov 02 15:15:00.331382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.331351    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"]
Nov 02 15:15:00.346285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.346250    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"]
Nov 02 15:15:00.373597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.373561    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd\" (UID: \"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"
Nov 02 15:15:00.373771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.373630    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mnrg\" (UniqueName: \"kubernetes.io/projected/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-kube-api-access-9mnrg\") pod \"maas-api-key-cleanup-29893875-jnj47\" (UID: \"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"
Nov 02 15:15:00.373771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.373659    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305f5164-3c9f-452d-a175-63c432126262-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb\" (UID: \"305f5164-3c9f-452d-a175-63c432126262\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"
Nov 02 15:15:00.373771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.373683    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znj2v\" (UniqueName: \"kubernetes.io/projected/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-kube-api-access-znj2v\") pod \"maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p\" (UID: \"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"
Nov 02 15:15:00.373771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.373749    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead7047e-5895-4773-a265-97e83d2b56b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g\" (UID: \"ead7047e-5895-4773-a265-97e83d2b56b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"
Nov 02 15:15:00.373990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.373775    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7lm\" (UniqueName: \"kubernetes.io/projected/ead7047e-5895-4773-a265-97e83d2b56b9-kube-api-access-th7lm\") pod \"maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g\" (UID: \"ead7047e-5895-4773-a265-97e83d2b56b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"
Nov 02 15:15:00.374053 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.374034    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwq7n\" (UniqueName: \"kubernetes.io/projected/305f5164-3c9f-452d-a175-63c432126262-kube-api-access-gwq7n\") pod \"maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb\" (UID: \"305f5164-3c9f-452d-a175-63c432126262\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"
Nov 02 15:15:00.374427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.374400    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd\" (UID: \"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"
Nov 02 15:15:00.374566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.374443    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p\" (UID: \"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"
Nov 02 15:15:00.374566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.374478    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9vs\" (UniqueName: \"kubernetes.io/projected/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-kube-api-access-2q9vs\") pod \"maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd\" (UID: \"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"
Nov 02 15:15:00.374566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.374521    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893875-jnj47\" (UID: \"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"
Nov 02 15:15:00.375452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.374595    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305f5164-3c9f-452d-a175-63c432126262-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb\" (UID: \"305f5164-3c9f-452d-a175-63c432126262\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"
Nov 02 15:15:00.375452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.375208    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893875-jnj47\" (UID: \"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"
Nov 02 15:15:00.375452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.375331    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p\" (UID: \"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"
Nov 02 15:15:00.394142 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.394112    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"]
Nov 02 15:15:00.443348 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.443243    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwq7n\" (UniqueName: \"kubernetes.io/projected/305f5164-3c9f-452d-a175-63c432126262-kube-api-access-gwq7n\") pod \"maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb\" (UID: \"305f5164-3c9f-452d-a175-63c432126262\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"
Nov 02 15:15:00.449001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.448976    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9vs\" (UniqueName: \"kubernetes.io/projected/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-kube-api-access-2q9vs\") pod \"maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd\" (UID: \"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"
Nov 02 15:15:00.449136 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.449024    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mnrg\" (UniqueName: \"kubernetes.io/projected/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-kube-api-access-9mnrg\") pod \"maas-api-key-cleanup-29893875-jnj47\" (UID: \"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"
Nov 02 15:15:00.450501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.450480    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znj2v\" (UniqueName: \"kubernetes.io/projected/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-kube-api-access-znj2v\") pod \"maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p\" (UID: \"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"
Nov 02 15:15:00.475979 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.475950    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead7047e-5895-4773-a265-97e83d2b56b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g\" (UID: \"ead7047e-5895-4773-a265-97e83d2b56b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"
Nov 02 15:15:00.476146 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.475982    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7lm\" (UniqueName: \"kubernetes.io/projected/ead7047e-5895-4773-a265-97e83d2b56b9-kube-api-access-th7lm\") pod \"maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g\" (UID: \"ead7047e-5895-4773-a265-97e83d2b56b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"
Nov 02 15:15:00.476670 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.476641    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead7047e-5895-4773-a265-97e83d2b56b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g\" (UID: \"ead7047e-5895-4773-a265-97e83d2b56b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"
Nov 02 15:15:00.512208 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.512173    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"
Nov 02 15:15:00.519798 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.519776    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"
Nov 02 15:15:00.525494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.525286    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"
Nov 02 15:15:00.537832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.537797    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7lm\" (UniqueName: \"kubernetes.io/projected/ead7047e-5895-4773-a265-97e83d2b56b9-kube-api-access-th7lm\") pod \"maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g\" (UID: \"ead7047e-5895-4773-a265-97e83d2b56b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"
Nov 02 15:15:00.556474 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.556443    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"
Nov 02 15:15:00.573759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.573536    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"
Nov 02 15:15:00.915764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.915736    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"]
Nov 02 15:15:00.915924 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:15:00.915897    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9656f1_4b62_4ca5_b59b_e1e22e588c9d.slice/crio-3b57a9f0b408f9bad67d8c8a28a652994eae9fdcaf645d2861e566006b06c18e WatchSource:0}: Error finding container 3b57a9f0b408f9bad67d8c8a28a652994eae9fdcaf645d2861e566006b06c18e: Status 404 returned error can't find the container with id 3b57a9f0b408f9bad67d8c8a28a652994eae9fdcaf645d2861e566006b06c18e
Nov 02 15:15:00.968025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.967984    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"]
Nov 02 15:15:00.968025 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:15:00.968006    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305f5164_3c9f_452d_a175_63c432126262.slice/crio-c7485af3a49e58483f149ff36ed90dfa568f1f10b66f2e8326e88505af60ad1c WatchSource:0}: Error finding container c7485af3a49e58483f149ff36ed90dfa568f1f10b66f2e8326e88505af60ad1c: Status 404 returned error can't find the container with id c7485af3a49e58483f149ff36ed90dfa568f1f10b66f2e8326e88505af60ad1c
Nov 02 15:15:00.973931 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:00.973911    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"]
Nov 02 15:15:01.013919 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:15:01.013892    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3dc5bc_289b_48f5_bdeb_d45d10b3a36e.slice/crio-e7ce9300ba5fbda1aa50c4f48531c0573261c2254550a7a3aa386420d754fd8c WatchSource:0}: Error finding container e7ce9300ba5fbda1aa50c4f48531c0573261c2254550a7a3aa386420d754fd8c: Status 404 returned error can't find the container with id e7ce9300ba5fbda1aa50c4f48531c0573261c2254550a7a3aa386420d754fd8c
Nov 02 15:15:01.034888 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:15:01.034859    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c75cc1_a1bf_4859_b1ca_10bb6ad853ed.slice/crio-64910720c59f701484800c47d73f470e8dbb96a28bc9dac642899c4abb7121e1 WatchSource:0}: Error finding container 64910720c59f701484800c47d73f470e8dbb96a28bc9dac642899c4abb7121e1: Status 404 returned error can't find the container with id 64910720c59f701484800c47d73f470e8dbb96a28bc9dac642899c4abb7121e1
Nov 02 15:15:01.035038 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.034869    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"]
Nov 02 15:15:01.046702 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.046672    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"]
Nov 02 15:15:01.064166 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:15:01.064139    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead7047e_5895_4773_a265_97e83d2b56b9.slice/crio-80f723ca2be92eb462657dea5404a704b6d9ed36542303c57abac8fb289dd54d WatchSource:0}: Error finding container 80f723ca2be92eb462657dea5404a704b6d9ed36542303c57abac8fb289dd54d: Status 404 returned error can't find the container with id 80f723ca2be92eb462657dea5404a704b6d9ed36542303c57abac8fb289dd54d
Nov 02 15:15:01.546978 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.546930    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd" event={"ID":"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e","Type":"ContainerStarted","Data":"e7ce9300ba5fbda1aa50c4f48531c0573261c2254550a7a3aa386420d754fd8c"}
Nov 02 15:15:01.549070 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.549034    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47" event={"ID":"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d","Type":"ContainerStarted","Data":"e5d8c0246f77a832728ab60a4ce85f30484ff0f1b6aff6c8373b1308bce821ab"}
Nov 02 15:15:01.549226 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.549092    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47" event={"ID":"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d","Type":"ContainerStarted","Data":"3b57a9f0b408f9bad67d8c8a28a652994eae9fdcaf645d2861e566006b06c18e"}
Nov 02 15:15:01.550904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.550878    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g" event={"ID":"ead7047e-5895-4773-a265-97e83d2b56b9","Type":"ContainerStarted","Data":"b1e5edcaea7b7cbaa577a705154c1194c3c51f00ceea41769642d349d8b059a5"}
Nov 02 15:15:01.551018 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.550919    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g" event={"ID":"ead7047e-5895-4773-a265-97e83d2b56b9","Type":"ContainerStarted","Data":"80f723ca2be92eb462657dea5404a704b6d9ed36542303c57abac8fb289dd54d"}
Nov 02 15:15:01.552664 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.552637    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb" event={"ID":"305f5164-3c9f-452d-a175-63c432126262","Type":"ContainerStarted","Data":"9a86caaf5de615257626d88d2150d57b17b6882133ba656ff574706d07c23b12"}
Nov 02 15:15:01.552793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.552769    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb" event={"ID":"305f5164-3c9f-452d-a175-63c432126262","Type":"ContainerStarted","Data":"c7485af3a49e58483f149ff36ed90dfa568f1f10b66f2e8326e88505af60ad1c"}
Nov 02 15:15:01.554287 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.554266    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p" event={"ID":"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed","Type":"ContainerStarted","Data":"59ca15bf9eba6c2b1a3d255981a8e32d0c93c96f80deea0009161b2801588659"}
Nov 02 15:15:01.554429 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.554348    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p" event={"ID":"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed","Type":"ContainerStarted","Data":"64910720c59f701484800c47d73f470e8dbb96a28bc9dac642899c4abb7121e1"}
Nov 02 15:15:01.669607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.669539    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47" podStartSLOduration=1.669521221 podStartE2EDuration="1.669521221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:01.666104017 +0000 UTC m=+1109.258720838" watchObservedRunningTime="2026-11-02 15:15:01.669521221 +0000 UTC m=+1109.262138060"
Nov 02 15:15:01.804784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.804669    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p" podStartSLOduration=1.804653904 podStartE2EDuration="1.804653904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:01.80156679 +0000 UTC m=+1109.394183590" watchObservedRunningTime="2026-11-02 15:15:01.804653904 +0000 UTC m=+1109.397270702"
Nov 02 15:15:01.919987 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:01.919930    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g" podStartSLOduration=1.919915112 podStartE2EDuration="1.919915112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:01.916582276 +0000 UTC m=+1109.509199076" watchObservedRunningTime="2026-11-02 15:15:01.919915112 +0000 UTC m=+1109.512531934"
Nov 02 15:15:02.034674 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.034619    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb" podStartSLOduration=2.034604394 podStartE2EDuration="2.034604394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:02.032529694 +0000 UTC m=+1109.625146493" watchObservedRunningTime="2026-11-02 15:15:02.034604394 +0000 UTC m=+1109.627221192"
Nov 02 15:15:02.564389 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.564341    4891 generic.go:356] "Generic (PLEG): container finished" podID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" containerID="8410b6817713f1f933065603ac1e1f392006268e8accd3cb38054b1a6b849679" exitCode=60
Nov 02 15:15:02.564832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.564402    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd" event={"ID":"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e","Type":"ContainerDied","Data":"8410b6817713f1f933065603ac1e1f392006268e8accd3cb38054b1a6b849679"}
Nov 02 15:15:02.564832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.564745    4891 scope.go:122] "RemoveContainer" containerID="8410b6817713f1f933065603ac1e1f392006268e8accd3cb38054b1a6b849679"
Nov 02 15:15:02.566250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.566222    4891 generic.go:356] "Generic (PLEG): container finished" podID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" containerID="e5d8c0246f77a832728ab60a4ce85f30484ff0f1b6aff6c8373b1308bce821ab" exitCode=60
Nov 02 15:15:02.566381 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.566321    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47" event={"ID":"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d","Type":"ContainerDied","Data":"e5d8c0246f77a832728ab60a4ce85f30484ff0f1b6aff6c8373b1308bce821ab"}
Nov 02 15:15:02.566580 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.566554    4891 scope.go:122] "RemoveContainer" containerID="e5d8c0246f77a832728ab60a4ce85f30484ff0f1b6aff6c8373b1308bce821ab"
Nov 02 15:15:02.568162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.568142    4891 generic.go:356] "Generic (PLEG): container finished" podID="ead7047e-5895-4773-a265-97e83d2b56b9" containerID="b1e5edcaea7b7cbaa577a705154c1194c3c51f00ceea41769642d349d8b059a5" exitCode=60
Nov 02 15:15:02.568472 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.568376    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g" event={"ID":"ead7047e-5895-4773-a265-97e83d2b56b9","Type":"ContainerDied","Data":"b1e5edcaea7b7cbaa577a705154c1194c3c51f00ceea41769642d349d8b059a5"}
Nov 02 15:15:02.568472 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.568434    4891 scope.go:122] "RemoveContainer" containerID="b1e5edcaea7b7cbaa577a705154c1194c3c51f00ceea41769642d349d8b059a5"
Nov 02 15:15:02.570223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.570203    4891 generic.go:356] "Generic (PLEG): container finished" podID="305f5164-3c9f-452d-a175-63c432126262" containerID="9a86caaf5de615257626d88d2150d57b17b6882133ba656ff574706d07c23b12" exitCode=60
Nov 02 15:15:02.570320 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.570281    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb" event={"ID":"305f5164-3c9f-452d-a175-63c432126262","Type":"ContainerDied","Data":"9a86caaf5de615257626d88d2150d57b17b6882133ba656ff574706d07c23b12"}
Nov 02 15:15:02.570475 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.570459    4891 scope.go:122] "RemoveContainer" containerID="9a86caaf5de615257626d88d2150d57b17b6882133ba656ff574706d07c23b12"
Nov 02 15:15:02.571836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.571817    4891 generic.go:356] "Generic (PLEG): container finished" podID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" containerID="59ca15bf9eba6c2b1a3d255981a8e32d0c93c96f80deea0009161b2801588659" exitCode=60
Nov 02 15:15:02.571940 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.571843    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p" event={"ID":"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed","Type":"ContainerDied","Data":"59ca15bf9eba6c2b1a3d255981a8e32d0c93c96f80deea0009161b2801588659"}
Nov 02 15:15:02.572090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:02.572074    4891 scope.go:122] "RemoveContainer" containerID="59ca15bf9eba6c2b1a3d255981a8e32d0c93c96f80deea0009161b2801588659"
Nov 02 15:15:03.581405 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:03.581372    4891 generic.go:356] "Generic (PLEG): container finished" podID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" containerID="f5ab59e37b0a78b5d9ff208d1ac114ff0cc73b429ec87ec92cf1d9fb86fb627f" exitCode=60
Nov 02 15:15:03.581876 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:03.581459    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd" event={"ID":"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e","Type":"ContainerDied","Data":"f5ab59e37b0a78b5d9ff208d1ac114ff0cc73b429ec87ec92cf1d9fb86fb627f"}
Nov 02 15:15:03.581876 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:03.581530    4891 scope.go:122] "RemoveContainer" containerID="8410b6817713f1f933065603ac1e1f392006268e8accd3cb38054b1a6b849679"
Nov 02 15:15:03.581876 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:03.581817    4891 scope.go:122] "RemoveContainer" containerID="f5ab59e37b0a78b5d9ff208d1ac114ff0cc73b429ec87ec92cf1d9fb86fb627f"
Nov 02 15:15:03.582124 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:03.582101    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd_odh-ai-gateway-infra(be3dc5bc-289b-48f5-bdeb-d45d10b3a36e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e"
Nov 02 15:15:03.583462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:03.583440    4891 generic.go:356] "Generic (PLEG): container finished" podID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" containerID="f29d00315d107143115d0b4ac388624327532911684b4668e05b8e637a2334e6" exitCode=60
Nov 02 15:15:03.583568 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:03.583477    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47" event={"ID":"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d","Type":"ContainerDied","Data":"f29d00315d107143115d0b4ac388624327532911684b4668e05b8e637a2334e6"}
Nov 02 15:15:03.583817 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:03.583802    4891 scope.go:122] "RemoveContainer" containerID="f29d00315d107143115d0b4ac388624327532911684b4668e05b8e637a2334e6"
Nov 02 15:15:03.584085 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:03.584062    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893875-jnj47_odh-ai-gateway-infra(ea9656f1-4b62-4ca5-b59b-e1e22e588c9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d"
Nov 02 15:15:03.585480 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:03.585462    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g" event={"ID":"ead7047e-5895-4773-a265-97e83d2b56b9","Type":"ContainerStarted","Data":"7550e110b98243e49e03df6034f4993ba11adaf4478e5dc142b6d897d60dc530"}
Nov 02 15:15:03.587431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:03.587405    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb" event={"ID":"305f5164-3c9f-452d-a175-63c432126262","Type":"ContainerStarted","Data":"1247793b44e4e0543830e727311bffab8434e50a96acdf864a1fcf70191728bc"}
Nov 02 15:15:03.589268 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:03.589250    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p" event={"ID":"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed","Type":"ContainerStarted","Data":"f55ea6200f59170df4820d93539df078f6f7c8af7f65d93b9cb57a62dee21ce7"}
Nov 02 15:15:03.859292 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:03.859270    4891 scope.go:122] "RemoveContainer" containerID="e5d8c0246f77a832728ab60a4ce85f30484ff0f1b6aff6c8373b1308bce821ab"
Nov 02 15:15:04.607436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.607399    4891 generic.go:356] "Generic (PLEG): container finished" podID="ead7047e-5895-4773-a265-97e83d2b56b9" containerID="7550e110b98243e49e03df6034f4993ba11adaf4478e5dc142b6d897d60dc530" exitCode=60
Nov 02 15:15:04.607951 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.607474    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g" event={"ID":"ead7047e-5895-4773-a265-97e83d2b56b9","Type":"ContainerDied","Data":"7550e110b98243e49e03df6034f4993ba11adaf4478e5dc142b6d897d60dc530"}
Nov 02 15:15:04.607951 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.607557    4891 scope.go:122] "RemoveContainer" containerID="b1e5edcaea7b7cbaa577a705154c1194c3c51f00ceea41769642d349d8b059a5"
Nov 02 15:15:04.607951 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.607755    4891 scope.go:122] "RemoveContainer" containerID="7550e110b98243e49e03df6034f4993ba11adaf4478e5dc142b6d897d60dc530"
Nov 02 15:15:04.608170 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:04.608093    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g_odh-ai-gateway-infra(ead7047e-5895-4773-a265-97e83d2b56b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g" podUID="ead7047e-5895-4773-a265-97e83d2b56b9"
Nov 02 15:15:04.609754 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.609732    4891 generic.go:356] "Generic (PLEG): container finished" podID="305f5164-3c9f-452d-a175-63c432126262" containerID="1247793b44e4e0543830e727311bffab8434e50a96acdf864a1fcf70191728bc" exitCode=60
Nov 02 15:15:04.609861 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.609810    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb" event={"ID":"305f5164-3c9f-452d-a175-63c432126262","Type":"ContainerDied","Data":"1247793b44e4e0543830e727311bffab8434e50a96acdf864a1fcf70191728bc"}
Nov 02 15:15:04.610042 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.609977    4891 scope.go:122] "RemoveContainer" containerID="1247793b44e4e0543830e727311bffab8434e50a96acdf864a1fcf70191728bc"
Nov 02 15:15:04.610260 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:04.610240    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb_odh-ai-gateway-infra(305f5164-3c9f-452d-a175-63c432126262)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb" podUID="305f5164-3c9f-452d-a175-63c432126262"
Nov 02 15:15:04.611985 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.611961    4891 generic.go:356] "Generic (PLEG): container finished" podID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" containerID="f55ea6200f59170df4820d93539df078f6f7c8af7f65d93b9cb57a62dee21ce7" exitCode=60
Nov 02 15:15:04.612071 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.612041    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p" event={"ID":"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed","Type":"ContainerDied","Data":"f55ea6200f59170df4820d93539df078f6f7c8af7f65d93b9cb57a62dee21ce7"}
Nov 02 15:15:04.612199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.612184    4891 scope.go:122] "RemoveContainer" containerID="f55ea6200f59170df4820d93539df078f6f7c8af7f65d93b9cb57a62dee21ce7"
Nov 02 15:15:04.612442 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:04.612422    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p_odh-ai-gateway-infra(79c75cc1-a1bf-4859-b1ca-10bb6ad853ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed"
Nov 02 15:15:04.614059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.614042    4891 scope.go:122] "RemoveContainer" containerID="f5ab59e37b0a78b5d9ff208d1ac114ff0cc73b429ec87ec92cf1d9fb86fb627f"
Nov 02 15:15:04.614206 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:04.614189    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd_odh-ai-gateway-infra(be3dc5bc-289b-48f5-bdeb-d45d10b3a36e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e"
Nov 02 15:15:04.615555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.615539    4891 scope.go:122] "RemoveContainer" containerID="f29d00315d107143115d0b4ac388624327532911684b4668e05b8e637a2334e6"
Nov 02 15:15:04.615827 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:04.615801    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893875-jnj47_odh-ai-gateway-infra(ea9656f1-4b62-4ca5-b59b-e1e22e588c9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d"
Nov 02 15:15:04.621557 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.621541    4891 scope.go:122] "RemoveContainer" containerID="9a86caaf5de615257626d88d2150d57b17b6882133ba656ff574706d07c23b12"
Nov 02 15:15:04.635129 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:04.635104    4891 scope.go:122] "RemoveContainer" containerID="59ca15bf9eba6c2b1a3d255981a8e32d0c93c96f80deea0009161b2801588659"
Nov 02 15:15:05.627440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:05.627411    4891 scope.go:122] "RemoveContainer" containerID="7550e110b98243e49e03df6034f4993ba11adaf4478e5dc142b6d897d60dc530"
Nov 02 15:15:05.627884 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:05.627727    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g_odh-ai-gateway-infra(ead7047e-5895-4773-a265-97e83d2b56b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g" podUID="ead7047e-5895-4773-a265-97e83d2b56b9"
Nov 02 15:15:05.629044 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:05.629019    4891 scope.go:122] "RemoveContainer" containerID="1247793b44e4e0543830e727311bffab8434e50a96acdf864a1fcf70191728bc"
Nov 02 15:15:05.629373 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:05.629351    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb_odh-ai-gateway-infra(305f5164-3c9f-452d-a175-63c432126262)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb" podUID="305f5164-3c9f-452d-a175-63c432126262"
Nov 02 15:15:05.630633 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:05.630615    4891 scope.go:122] "RemoveContainer" containerID="f55ea6200f59170df4820d93539df078f6f7c8af7f65d93b9cb57a62dee21ce7"
Nov 02 15:15:05.630814 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:05.630798    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p_odh-ai-gateway-infra(79c75cc1-a1bf-4859-b1ca-10bb6ad853ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed"
Nov 02 15:15:14.274354 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:14.274237    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:15:14.515247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:14.515216    4891 scope.go:122] "RemoveContainer" containerID="1247793b44e4e0543830e727311bffab8434e50a96acdf864a1fcf70191728bc"
Nov 02 15:15:15.514922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:15.514890    4891 scope.go:122] "RemoveContainer" containerID="7550e110b98243e49e03df6034f4993ba11adaf4478e5dc142b6d897d60dc530"
Nov 02 15:15:15.707181 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:15.707096    4891 generic.go:356] "Generic (PLEG): container finished" podID="305f5164-3c9f-452d-a175-63c432126262" containerID="f5c739893bb98739c3dd2394c61f6006d37f9e7618506f33c6d59ef80c81a8d7" exitCode=60
Nov 02 15:15:15.707375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:15.707180    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb" event={"ID":"305f5164-3c9f-452d-a175-63c432126262","Type":"ContainerDied","Data":"f5c739893bb98739c3dd2394c61f6006d37f9e7618506f33c6d59ef80c81a8d7"}
Nov 02 15:15:15.707375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:15.707255    4891 scope.go:122] "RemoveContainer" containerID="1247793b44e4e0543830e727311bffab8434e50a96acdf864a1fcf70191728bc"
Nov 02 15:15:15.707743 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:15.707710    4891 scope.go:122] "RemoveContainer" containerID="f5c739893bb98739c3dd2394c61f6006d37f9e7618506f33c6d59ef80c81a8d7"
Nov 02 15:15:15.708059 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:15.708040    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb_odh-ai-gateway-infra(305f5164-3c9f-452d-a175-63c432126262)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb" podUID="305f5164-3c9f-452d-a175-63c432126262"
Nov 02 15:15:16.515270 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:16.515234    4891 scope.go:122] "RemoveContainer" containerID="f5ab59e37b0a78b5d9ff208d1ac114ff0cc73b429ec87ec92cf1d9fb86fb627f"
Nov 02 15:15:16.515757 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:16.515492    4891 scope.go:122] "RemoveContainer" containerID="f55ea6200f59170df4820d93539df078f6f7c8af7f65d93b9cb57a62dee21ce7"
Nov 02 15:15:16.716939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:16.716853    4891 generic.go:356] "Generic (PLEG): container finished" podID="ead7047e-5895-4773-a265-97e83d2b56b9" containerID="6ec1665c41b7691a583e8e17352205e5a1063e827e0e8afa943ac5cc1ba0c628" exitCode=60
Nov 02 15:15:16.716939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:16.716895    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g" event={"ID":"ead7047e-5895-4773-a265-97e83d2b56b9","Type":"ContainerDied","Data":"6ec1665c41b7691a583e8e17352205e5a1063e827e0e8afa943ac5cc1ba0c628"}
Nov 02 15:15:16.717178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:16.716955    4891 scope.go:122] "RemoveContainer" containerID="7550e110b98243e49e03df6034f4993ba11adaf4478e5dc142b6d897d60dc530"
Nov 02 15:15:16.717363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:16.717345    4891 scope.go:122] "RemoveContainer" containerID="6ec1665c41b7691a583e8e17352205e5a1063e827e0e8afa943ac5cc1ba0c628"
Nov 02 15:15:16.717743 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:16.717718    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g_odh-ai-gateway-infra(ead7047e-5895-4773-a265-97e83d2b56b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g" podUID="ead7047e-5895-4773-a265-97e83d2b56b9"
Nov 02 15:15:16.835546 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:16.835512    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"]
Nov 02 15:15:17.005629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.005600    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"
Nov 02 15:15:17.181413 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.181379    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305f5164-3c9f-452d-a175-63c432126262-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305f5164-3c9f-452d-a175-63c432126262-maas-api-service-ca\") pod \"305f5164-3c9f-452d-a175-63c432126262\" (UID: \"305f5164-3c9f-452d-a175-63c432126262\") "
Nov 02 15:15:17.181581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.181422    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305f5164-3c9f-452d-a175-63c432126262-kube-api-access-gwq7n\" (UniqueName: \"kubernetes.io/projected/305f5164-3c9f-452d-a175-63c432126262-kube-api-access-gwq7n\") pod \"305f5164-3c9f-452d-a175-63c432126262\" (UID: \"305f5164-3c9f-452d-a175-63c432126262\") "
Nov 02 15:15:17.181819 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.181784    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305f5164-3c9f-452d-a175-63c432126262-maas-api-service-ca" pod "305f5164-3c9f-452d-a175-63c432126262" (UID: "305f5164-3c9f-452d-a175-63c432126262"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:17.181990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.181977    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305f5164-3c9f-452d-a175-63c432126262-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:17.183574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.183553    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305f5164-3c9f-452d-a175-63c432126262-kube-api-access-gwq7n" pod "305f5164-3c9f-452d-a175-63c432126262" (UID: "305f5164-3c9f-452d-a175-63c432126262"). InnerVolumeSpecName "kube-api-access-gwq7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:17.283111 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.283075    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwq7n\" (UniqueName: \"kubernetes.io/projected/305f5164-3c9f-452d-a175-63c432126262-kube-api-access-gwq7n\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:17.515766 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.515737    4891 scope.go:122] "RemoveContainer" containerID="f29d00315d107143115d0b4ac388624327532911684b4668e05b8e637a2334e6"
Nov 02 15:15:17.729371 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.729334    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb" event={"ID":"305f5164-3c9f-452d-a175-63c432126262","Type":"ContainerDied","Data":"c7485af3a49e58483f149ff36ed90dfa568f1f10b66f2e8326e88505af60ad1c"}
Nov 02 15:15:17.729563 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.729418    4891 scope.go:122] "RemoveContainer" containerID="f5c739893bb98739c3dd2394c61f6006d37f9e7618506f33c6d59ef80c81a8d7"
Nov 02 15:15:17.729563 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.729347    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"
Nov 02 15:15:17.731535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.731510    4891 generic.go:356] "Generic (PLEG): container finished" podID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" containerID="5cae7f62654ae9d127b43a4adb89d0d866f638a2551ab865ba6e33fa346fe760" exitCode=60
Nov 02 15:15:17.731671 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.731580    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p" event={"ID":"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed","Type":"ContainerDied","Data":"5cae7f62654ae9d127b43a4adb89d0d866f638a2551ab865ba6e33fa346fe760"}
Nov 02 15:15:17.732023 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.732004    4891 scope.go:122] "RemoveContainer" containerID="5cae7f62654ae9d127b43a4adb89d0d866f638a2551ab865ba6e33fa346fe760"
Nov 02 15:15:17.732363 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:17.732305    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p_odh-ai-gateway-infra(79c75cc1-a1bf-4859-b1ca-10bb6ad853ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed"
Nov 02 15:15:17.737376 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.737355    4891 generic.go:356] "Generic (PLEG): container finished" podID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" containerID="792edb5402f56e04988633f335e77ba87cd3c54118eb17a265204ddc635c674e" exitCode=60
Nov 02 15:15:17.737498 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.737437    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd" event={"ID":"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e","Type":"ContainerDied","Data":"792edb5402f56e04988633f335e77ba87cd3c54118eb17a265204ddc635c674e"}
Nov 02 15:15:17.737787 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.737773    4891 scope.go:122] "RemoveContainer" containerID="792edb5402f56e04988633f335e77ba87cd3c54118eb17a265204ddc635c674e"
Nov 02 15:15:17.738034 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:17.738013    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd_odh-ai-gateway-infra(be3dc5bc-289b-48f5-bdeb-d45d10b3a36e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e"
Nov 02 15:15:17.741691 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.741675    4891 scope.go:122] "RemoveContainer" containerID="f55ea6200f59170df4820d93539df078f6f7c8af7f65d93b9cb57a62dee21ce7"
Nov 02 15:15:17.767000 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.766979    4891 scope.go:122] "RemoveContainer" containerID="f5ab59e37b0a78b5d9ff208d1ac114ff0cc73b429ec87ec92cf1d9fb86fb627f"
Nov 02 15:15:17.852182 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:17.852152    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"]
Nov 02 15:15:18.006936 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.006909    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"
Nov 02 15:15:18.132041 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.131957    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"]
Nov 02 15:15:18.136157 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.136131    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-019efe-29893875-xxscb"]
Nov 02 15:15:18.194441 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.194407    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead7047e-5895-4773-a265-97e83d2b56b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead7047e-5895-4773-a265-97e83d2b56b9-maas-api-service-ca\") pod \"ead7047e-5895-4773-a265-97e83d2b56b9\" (UID: \"ead7047e-5895-4773-a265-97e83d2b56b9\") "
Nov 02 15:15:18.194591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.194476    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead7047e-5895-4773-a265-97e83d2b56b9-kube-api-access-th7lm\" (UniqueName: \"kubernetes.io/projected/ead7047e-5895-4773-a265-97e83d2b56b9-kube-api-access-th7lm\") pod \"ead7047e-5895-4773-a265-97e83d2b56b9\" (UID: \"ead7047e-5895-4773-a265-97e83d2b56b9\") "
Nov 02 15:15:18.194781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.194755    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead7047e-5895-4773-a265-97e83d2b56b9-maas-api-service-ca" pod "ead7047e-5895-4773-a265-97e83d2b56b9" (UID: "ead7047e-5895-4773-a265-97e83d2b56b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:18.196568 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.196548    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead7047e-5895-4773-a265-97e83d2b56b9-kube-api-access-th7lm" pod "ead7047e-5895-4773-a265-97e83d2b56b9" (UID: "ead7047e-5895-4773-a265-97e83d2b56b9"). InnerVolumeSpecName "kube-api-access-th7lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:18.295728 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.295683    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead7047e-5895-4773-a265-97e83d2b56b9-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:18.295728 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.295724    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7lm\" (UniqueName: \"kubernetes.io/projected/ead7047e-5895-4773-a265-97e83d2b56b9-kube-api-access-th7lm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:18.522545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.522513    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305f5164-3c9f-452d-a175-63c432126262" path="/var/lib/kubelet/pods/305f5164-3c9f-452d-a175-63c432126262/volumes"
Nov 02 15:15:18.746502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.746420    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"
Nov 02 15:15:18.746502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.746419    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g" event={"ID":"ead7047e-5895-4773-a265-97e83d2b56b9","Type":"ContainerDied","Data":"80f723ca2be92eb462657dea5404a704b6d9ed36542303c57abac8fb289dd54d"}
Nov 02 15:15:18.746720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.746557    4891 scope.go:122] "RemoveContainer" containerID="6ec1665c41b7691a583e8e17352205e5a1063e827e0e8afa943ac5cc1ba0c628"
Nov 02 15:15:18.753326 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.753290    4891 generic.go:356] "Generic (PLEG): container finished" podID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" containerID="d14f9021d9e2d7ed242ff9dbcaeae6419201bb2f36ad84c2f0d631e93dcc67e0" exitCode=60
Nov 02 15:15:18.753450 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.753336    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47" event={"ID":"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d","Type":"ContainerDied","Data":"d14f9021d9e2d7ed242ff9dbcaeae6419201bb2f36ad84c2f0d631e93dcc67e0"}
Nov 02 15:15:18.753718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.753692    4891 scope.go:122] "RemoveContainer" containerID="d14f9021d9e2d7ed242ff9dbcaeae6419201bb2f36ad84c2f0d631e93dcc67e0"
Nov 02 15:15:18.753978 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:18.753956    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893875-jnj47_odh-ai-gateway-infra(ea9656f1-4b62-4ca5-b59b-e1e22e588c9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d"
Nov 02 15:15:18.758059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.758043    4891 scope.go:122] "RemoveContainer" containerID="f29d00315d107143115d0b4ac388624327532911684b4668e05b8e637a2334e6"
Nov 02 15:15:18.912247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:18.912219    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"]
Nov 02 15:15:19.028906 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.028873    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"]
Nov 02 15:15:19.033192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.033164    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-c18e79-29893875-qmw6g"]
Nov 02 15:15:19.037081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.037053    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"]
Nov 02 15:15:19.072704 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.072677    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"
Nov 02 15:15:19.191799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.191775    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"
Nov 02 15:15:19.207381 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.207354    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-kube-api-access-2q9vs\" (UniqueName: \"kubernetes.io/projected/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-kube-api-access-2q9vs\") pod \"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e\" (UID: \"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e\") "
Nov 02 15:15:19.207485 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.207401    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-maas-api-service-ca\") pod \"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e\" (UID: \"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e\") "
Nov 02 15:15:19.207758 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.207738    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-maas-api-service-ca" pod "be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" (UID: "be3dc5bc-289b-48f5-bdeb-d45d10b3a36e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:19.209359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.209338    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-kube-api-access-2q9vs" pod "be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" (UID: "be3dc5bc-289b-48f5-bdeb-d45d10b3a36e"). InnerVolumeSpecName "kube-api-access-2q9vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:19.308749 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.308671    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-kube-api-access-znj2v\" (UniqueName: \"kubernetes.io/projected/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-kube-api-access-znj2v\") pod \"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed\" (UID: \"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed\") "
Nov 02 15:15:19.308890 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.308815    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-maas-api-service-ca\") pod \"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed\" (UID: \"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed\") "
Nov 02 15:15:19.309088 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.309077    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q9vs\" (UniqueName: \"kubernetes.io/projected/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-kube-api-access-2q9vs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:19.309136 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.309091    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:19.309185 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.309164    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-maas-api-service-ca" pod "79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" (UID: "79c75cc1-a1bf-4859-b1ca-10bb6ad853ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:19.310892 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.310863    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-kube-api-access-znj2v" pod "79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" (UID: "79c75cc1-a1bf-4859-b1ca-10bb6ad853ed"). InnerVolumeSpecName "kube-api-access-znj2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:19.414382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.414342    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:19.414382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.414382    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znj2v\" (UniqueName: \"kubernetes.io/projected/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed-kube-api-access-znj2v\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:19.764386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.764296    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"
Nov 02 15:15:19.764836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.764290    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p" event={"ID":"79c75cc1-a1bf-4859-b1ca-10bb6ad853ed","Type":"ContainerDied","Data":"64910720c59f701484800c47d73f470e8dbb96a28bc9dac642899c4abb7121e1"}
Nov 02 15:15:19.764836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.764449    4891 scope.go:122] "RemoveContainer" containerID="5cae7f62654ae9d127b43a4adb89d0d866f638a2551ab865ba6e33fa346fe760"
Nov 02 15:15:19.765779 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.765761    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd" event={"ID":"be3dc5bc-289b-48f5-bdeb-d45d10b3a36e","Type":"ContainerDied","Data":"e7ce9300ba5fbda1aa50c4f48531c0573261c2254550a7a3aa386420d754fd8c"}
Nov 02 15:15:19.765854 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.765778    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"
Nov 02 15:15:19.781367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.777748    4891 scope.go:122] "RemoveContainer" containerID="792edb5402f56e04988633f335e77ba87cd3c54118eb17a265204ddc635c674e"
Nov 02 15:15:19.927278 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.927241    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"]
Nov 02 15:15:19.933117 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.933089    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"]
Nov 02 15:15:19.945156 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:19.943869    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-42555d-29893875-jhmqd"]
Nov 02 15:15:20.055402 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.055373    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"]
Nov 02 15:15:20.063018 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.062995    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893875-zn52p"]
Nov 02 15:15:20.069584 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.069565    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"
Nov 02 15:15:20.121421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.121394    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-maas-api-service-ca\") pod \"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d\" (UID: \"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d\") "
Nov 02 15:15:20.121593 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.121438    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-kube-api-access-9mnrg\" (UniqueName: \"kubernetes.io/projected/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-kube-api-access-9mnrg\") pod \"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d\" (UID: \"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d\") "
Nov 02 15:15:20.121818 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.121795    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-maas-api-service-ca" pod "ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" (UID: "ea9656f1-4b62-4ca5-b59b-e1e22e588c9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:20.121980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.121966    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:20.123593 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.123569    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-kube-api-access-9mnrg" pod "ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" (UID: "ea9656f1-4b62-4ca5-b59b-e1e22e588c9d"). InnerVolumeSpecName "kube-api-access-9mnrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:20.223665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.223617    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mnrg\" (UniqueName: \"kubernetes.io/projected/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d-kube-api-access-9mnrg\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:20.523300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.523270    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" path="/var/lib/kubelet/pods/79c75cc1-a1bf-4859-b1ca-10bb6ad853ed/volumes"
Nov 02 15:15:20.523787 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.523772    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" path="/var/lib/kubelet/pods/be3dc5bc-289b-48f5-bdeb-d45d10b3a36e/volumes"
Nov 02 15:15:20.524248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.524232    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead7047e-5895-4773-a265-97e83d2b56b9" path="/var/lib/kubelet/pods/ead7047e-5895-4773-a265-97e83d2b56b9/volumes"
Nov 02 15:15:20.780880 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.780794    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47" event={"ID":"ea9656f1-4b62-4ca5-b59b-e1e22e588c9d","Type":"ContainerDied","Data":"3b57a9f0b408f9bad67d8c8a28a652994eae9fdcaf645d2861e566006b06c18e"}
Nov 02 15:15:20.780880 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.780873    4891 scope.go:122] "RemoveContainer" containerID="d14f9021d9e2d7ed242ff9dbcaeae6419201bb2f36ad84c2f0d631e93dcc67e0"
Nov 02 15:15:20.781407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.780813    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"
Nov 02 15:15:20.948833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.948795    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"]
Nov 02 15:15:20.955156 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:20.955126    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-jnj47"]
Nov 02 15:15:22.523738 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:22.523707    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" path="/var/lib/kubelet/pods/ea9656f1-4b62-4ca5-b59b-e1e22e588c9d/volumes"
Nov 02 15:15:24.357684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.357651    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"]
Nov 02 15:15:24.358194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358118    4891 cpu_manager.go:414] "containerMap: removing container" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" containerName="cleanup"
Nov 02 15:15:24.358194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358130    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" containerName="cleanup"
Nov 02 15:15:24.358194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358144    4891 cpu_manager.go:414] "containerMap: removing container" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" containerName="cleanup"
Nov 02 15:15:24.358194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358149    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" containerName="cleanup"
Nov 02 15:15:24.358194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358163    4891 cpu_manager.go:414] "containerMap: removing container" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" containerName="cleanup"
Nov 02 15:15:24.358194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358168    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" containerName="cleanup"
Nov 02 15:15:24.358194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358179    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" containerName="cleanup"
Nov 02 15:15:24.358194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358185    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" containerName="cleanup"
Nov 02 15:15:24.358194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358193    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" containerName="cleanup"
Nov 02 15:15:24.358194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358197    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358208    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ead7047e-5895-4773-a265-97e83d2b56b9" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358213    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead7047e-5895-4773-a265-97e83d2b56b9" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358222    4891 cpu_manager.go:414] "containerMap: removing container" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358226    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358235    4891 cpu_manager.go:414] "containerMap: removing container" podUID="305f5164-3c9f-452d-a175-63c432126262" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358240    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305f5164-3c9f-452d-a175-63c432126262" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358246    4891 cpu_manager.go:414] "containerMap: removing container" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358250    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358257    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ead7047e-5895-4773-a265-97e83d2b56b9" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358261    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead7047e-5895-4773-a265-97e83d2b56b9" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358268    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ead7047e-5895-4773-a265-97e83d2b56b9" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358272    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead7047e-5895-4773-a265-97e83d2b56b9" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358279    4891 cpu_manager.go:414] "containerMap: removing container" podUID="305f5164-3c9f-452d-a175-63c432126262" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358283    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305f5164-3c9f-452d-a175-63c432126262" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358298    4891 cpu_manager.go:414] "containerMap: removing container" podUID="305f5164-3c9f-452d-a175-63c432126262" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358302    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305f5164-3c9f-452d-a175-63c432126262" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358400    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead7047e-5895-4773-a265-97e83d2b56b9" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358408    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358414    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358420    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="305f5164-3c9f-452d-a175-63c432126262" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358426    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358435    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead7047e-5895-4773-a265-97e83d2b56b9" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358441    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead7047e-5895-4773-a265-97e83d2b56b9" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358446    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358451    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="305f5164-3c9f-452d-a175-63c432126262" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358457    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358465    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358472    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" containerName="cleanup"
Nov 02 15:15:24.358715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.358478    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" containerName="cleanup"
Nov 02 15:15:24.363584 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.363562    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:24.419399 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.419367    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"]
Nov 02 15:15:24.463950 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.463917    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:15:24.464133 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.463965    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:15:24.464133 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.463996    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:15:24.465156 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.465140    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:15:24.466673 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.466650    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt98g\" (UniqueName: \"kubernetes.io/projected/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-kube-api-access-tt98g\") pod \"maas-api-revoke-sub-a5a39269-j2glj\" (UID: \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:24.466773 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.466716    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a39269-j2glj\" (UID: \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:24.466871 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.466856    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a39269-j2glj\" (UID: \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:24.567491 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.567457    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a39269-j2glj\" (UID: \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:24.567651 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.567538    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt98g\" (UniqueName: \"kubernetes.io/projected/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-kube-api-access-tt98g\") pod \"maas-api-revoke-sub-a5a39269-j2glj\" (UID: \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:24.567651 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.567575    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a39269-j2glj\" (UID: \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:24.568223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.568206    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a39269-j2glj\" (UID: \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:24.569859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.569836    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a39269-j2glj\" (UID: \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:24.628546 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.628479    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt98g\" (UniqueName: \"kubernetes.io/projected/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-kube-api-access-tt98g\") pod \"maas-api-revoke-sub-a5a39269-j2glj\" (UID: \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:24.677144 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:24.677113    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:25.063819 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:25.063458    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"]
Nov 02 15:15:25.866188 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:25.865819    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj" event={"ID":"af7f96e3-d48c-4ee2-a6d9-81677ddf708a","Type":"ContainerStarted","Data":"efc6014e59878cc5af346b60adf3f1a1bae84f6e2107b1c20a689921ac836835"}
Nov 02 15:15:25.866188 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:25.865885    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj" event={"ID":"af7f96e3-d48c-4ee2-a6d9-81677ddf708a","Type":"ContainerStarted","Data":"8e40ad23c0183ca2fc24b9820cabda93ec1308817bd3b5664274d2484a9f3db7"}
Nov 02 15:15:25.981701 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:25.981620    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj" podStartSLOduration=1.981581316 podStartE2EDuration="1.981581316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:25.981203281 +0000 UTC m=+1133.573820078" watchObservedRunningTime="2026-11-02 15:15:25.981581316 +0000 UTC m=+1133.574198115"
Nov 02 15:15:26.876152 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:26.876119    4891 generic.go:356] "Generic (PLEG): container finished" podID="af7f96e3-d48c-4ee2-a6d9-81677ddf708a" containerID="efc6014e59878cc5af346b60adf3f1a1bae84f6e2107b1c20a689921ac836835" exitCode=0
Nov 02 15:15:26.876686 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:26.876213    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj" event={"ID":"af7f96e3-d48c-4ee2-a6d9-81677ddf708a","Type":"ContainerDied","Data":"efc6014e59878cc5af346b60adf3f1a1bae84f6e2107b1c20a689921ac836835"}
Nov 02 15:15:28.031623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.031597    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:28.110101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.110068    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-openshift-ca\") pod \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\" (UID: \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\") "
Nov 02 15:15:28.110281 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.110228    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-kube-api-access-tt98g\" (UniqueName: \"kubernetes.io/projected/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-kube-api-access-tt98g\") pod \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\" (UID: \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\") "
Nov 02 15:15:28.110370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.110354    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-xks-ca\") pod \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\" (UID: \"af7f96e3-d48c-4ee2-a6d9-81677ddf708a\") "
Nov 02 15:15:28.110483 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.110457    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-openshift-ca" pod "af7f96e3-d48c-4ee2-a6d9-81677ddf708a" (UID: "af7f96e3-d48c-4ee2-a6d9-81677ddf708a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:28.110826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.110804    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:28.112252 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.112228    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-kube-api-access-tt98g" pod "af7f96e3-d48c-4ee2-a6d9-81677ddf708a" (UID: "af7f96e3-d48c-4ee2-a6d9-81677ddf708a"). InnerVolumeSpecName "kube-api-access-tt98g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:28.112392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.112370    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-xks-ca" pod "af7f96e3-d48c-4ee2-a6d9-81677ddf708a" (UID: "af7f96e3-d48c-4ee2-a6d9-81677ddf708a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:15:28.213386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.213233    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt98g\" (UniqueName: \"kubernetes.io/projected/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-kube-api-access-tt98g\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:28.213386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.213272    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af7f96e3-d48c-4ee2-a6d9-81677ddf708a-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:28.893368 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.893252    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"
Nov 02 15:15:28.893368 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.893275    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj" event={"ID":"af7f96e3-d48c-4ee2-a6d9-81677ddf708a","Type":"ContainerDied","Data":"8e40ad23c0183ca2fc24b9820cabda93ec1308817bd3b5664274d2484a9f3db7"}
Nov 02 15:15:28.893368 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:28.893365    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e40ad23c0183ca2fc24b9820cabda93ec1308817bd3b5664274d2484a9f3db7"
Nov 02 15:15:44.275756 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:15:44.275712    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:15:52.276684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.276637    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"]
Nov 02 15:15:52.277791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.277761    4891 cpu_manager.go:414] "containerMap: removing container" podUID="af7f96e3-d48c-4ee2-a6d9-81677ddf708a" containerName="revoke-keys"
Nov 02 15:15:52.277791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.277790    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7f96e3-d48c-4ee2-a6d9-81677ddf708a" containerName="revoke-keys"
Nov 02 15:15:52.277980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.277820    4891 cpu_manager.go:414] "containerMap: removing container" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" containerName="cleanup"
Nov 02 15:15:52.277980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.277828    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c75cc1-a1bf-4859-b1ca-10bb6ad853ed" containerName="cleanup"
Nov 02 15:15:52.277980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.277871    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" containerName="cleanup"
Nov 02 15:15:52.277980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.277879    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9656f1-4b62-4ca5-b59b-e1e22e588c9d" containerName="cleanup"
Nov 02 15:15:52.278155 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.278105    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3dc5bc-289b-48f5-bdeb-d45d10b3a36e" containerName="cleanup"
Nov 02 15:15:52.278155 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.278119    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="305f5164-3c9f-452d-a175-63c432126262" containerName="cleanup"
Nov 02 15:15:52.278155 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.278141    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7f96e3-d48c-4ee2-a6d9-81677ddf708a" containerName="revoke-keys"
Nov 02 15:15:52.283633 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.283609    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:52.334562 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.334531    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"]
Nov 02 15:15:52.378860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.378829    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:15:52.378860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.378850    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:15:52.378860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.378850    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:15:52.385441 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.385419    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:15:52.386975 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.386952    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwmj\" (UniqueName: \"kubernetes.io/projected/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-kube-api-access-lzwmj\") pod \"maas-api-revoke-sub-b65040ec-jvlvb\" (UID: \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:52.387091 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.386988    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b65040ec-jvlvb\" (UID: \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:52.387091 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.387014    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b65040ec-jvlvb\" (UID: \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:52.488499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.488466    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwmj\" (UniqueName: \"kubernetes.io/projected/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-kube-api-access-lzwmj\") pod \"maas-api-revoke-sub-b65040ec-jvlvb\" (UID: \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:52.488499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.488503    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b65040ec-jvlvb\" (UID: \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:52.488770 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.488529    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b65040ec-jvlvb\" (UID: \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:52.489199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.489171    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b65040ec-jvlvb\" (UID: \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:52.491028 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.491008    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b65040ec-jvlvb\" (UID: \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:52.549055 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.548985    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwmj\" (UniqueName: \"kubernetes.io/projected/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-kube-api-access-lzwmj\") pod \"maas-api-revoke-sub-b65040ec-jvlvb\" (UID: \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:52.596841 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.596812    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:52.980826 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:15:52.980792    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3197e5_9b98_49e0_8c06_32bfd41a7fbe.slice/crio-84fd2262ad166b0628f1b55d2f82345d7ac6300089dbbbcd50462e5b52e3d3cc WatchSource:0}: Error finding container 84fd2262ad166b0628f1b55d2f82345d7ac6300089dbbbcd50462e5b52e3d3cc: Status 404 returned error can't find the container with id 84fd2262ad166b0628f1b55d2f82345d7ac6300089dbbbcd50462e5b52e3d3cc
Nov 02 15:15:52.980826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:52.980805    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"]
Nov 02 15:15:53.074215 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:53.074183    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb" event={"ID":"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe","Type":"ContainerStarted","Data":"84fd2262ad166b0628f1b55d2f82345d7ac6300089dbbbcd50462e5b52e3d3cc"}
Nov 02 15:15:54.083360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:54.083258    4891 generic.go:356] "Generic (PLEG): container finished" podID="0d3197e5-9b98-49e0-8c06-32bfd41a7fbe" containerID="f490cddd31b18874ae406beec6a6ba512797fb044dad97bcc3ddb870d915aa45" exitCode=0
Nov 02 15:15:54.083736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:54.083358    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb" event={"ID":"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe","Type":"ContainerDied","Data":"f490cddd31b18874ae406beec6a6ba512797fb044dad97bcc3ddb870d915aa45"}
Nov 02 15:15:55.064271 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:55.064235    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"]
Nov 02 15:15:55.113912 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:55.113877    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ca0ac4-54kpg"]
Nov 02 15:15:55.242158 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:55.242136    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:55.322784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:55.322705    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-kube-api-access-lzwmj\" (UniqueName: \"kubernetes.io/projected/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-kube-api-access-lzwmj\") pod \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\" (UID: \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\") "
Nov 02 15:15:55.322941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:55.322842    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-openshift-ca\") pod \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\" (UID: \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\") "
Nov 02 15:15:55.322941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:55.322892    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-xks-ca\") pod \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\" (UID: \"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe\") "
Nov 02 15:15:55.323247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:55.323217    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-openshift-ca" pod "0d3197e5-9b98-49e0-8c06-32bfd41a7fbe" (UID: "0d3197e5-9b98-49e0-8c06-32bfd41a7fbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:55.324972 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:55.324946    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-kube-api-access-lzwmj" pod "0d3197e5-9b98-49e0-8c06-32bfd41a7fbe" (UID: "0d3197e5-9b98-49e0-8c06-32bfd41a7fbe"). InnerVolumeSpecName "kube-api-access-lzwmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:55.325442 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:55.325419    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-xks-ca" pod "0d3197e5-9b98-49e0-8c06-32bfd41a7fbe" (UID: "0d3197e5-9b98-49e0-8c06-32bfd41a7fbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:15:55.424192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:55.424146    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:55.424192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:55.424188    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:55.424192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:55.424197    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzwmj\" (UniqueName: \"kubernetes.io/projected/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe-kube-api-access-lzwmj\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:15:56.101661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:56.101630    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"
Nov 02 15:15:56.101661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:56.101655    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb" event={"ID":"0d3197e5-9b98-49e0-8c06-32bfd41a7fbe","Type":"ContainerDied","Data":"84fd2262ad166b0628f1b55d2f82345d7ac6300089dbbbcd50462e5b52e3d3cc"}
Nov 02 15:15:56.101861 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:56.101725    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84fd2262ad166b0628f1b55d2f82345d7ac6300089dbbbcd50462e5b52e3d3cc"
Nov 02 15:15:56.523407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:56.523373    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc01975-8c7b-4518-a5db-ab7cc59e8b15" path="/var/lib/kubelet/pods/9fc01975-8c7b-4518-a5db-ab7cc59e8b15/volumes"
Nov 02 15:15:59.379677 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.379635    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"]
Nov 02 15:15:59.380150 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.380135    4891 cpu_manager.go:414] "containerMap: removing container" podUID="0d3197e5-9b98-49e0-8c06-32bfd41a7fbe" containerName="revoke-keys"
Nov 02 15:15:59.380194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.380151    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3197e5-9b98-49e0-8c06-32bfd41a7fbe" containerName="revoke-keys"
Nov 02 15:15:59.380261 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.380252    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3197e5-9b98-49e0-8c06-32bfd41a7fbe" containerName="revoke-keys"
Nov 02 15:15:59.384930 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.384911    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.442651 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.442617    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"]
Nov 02 15:15:59.467473 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.467441    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da5799a-02da-4d12-98ae-7eb568707007-tls-certs\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.467620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.467489    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-model-cache\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.467620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.467510    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mff\" (UniqueName: \"kubernetes.io/projected/4da5799a-02da-4d12-98ae-7eb568707007-kube-api-access-94mff\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.467787 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.467761    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-home\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.467839 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.467805    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-dshm\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.467993 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.467975    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-tmp-dir\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.486982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.486951    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-019efe\"/\"distinct-2-w1-019efe-kserve-self-signed-certs\""
Nov 02 15:15:59.569511 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.569476    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-tmp-dir\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.569720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.569528    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da5799a-02da-4d12-98ae-7eb568707007-tls-certs\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.569720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.569563    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-model-cache\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.569720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.569581    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mff\" (UniqueName: \"kubernetes.io/projected/4da5799a-02da-4d12-98ae-7eb568707007-kube-api-access-94mff\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.569720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.569700    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-home\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.569954 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.569726    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-dshm\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.570011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.569981    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-tmp-dir\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.570097 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.570072    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-model-cache\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.570281 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.570253    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-home\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.572212 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.572192    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-dshm\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.572501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.572483    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da5799a-02da-4d12-98ae-7eb568707007-tls-certs\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.631036 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.630965    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mff\" (UniqueName: \"kubernetes.io/projected/4da5799a-02da-4d12-98ae-7eb568707007-kube-api-access-94mff\") pod \"distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") " pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:15:59.697871 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:15:59.697828    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:16:00.085571 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:16:00.085543    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4da5799a_02da_4d12_98ae_7eb568707007.slice/crio-bb2eaf51ed39abb2048c19a31a7c222efa081361f95d07904d4f1bc7ac967009 WatchSource:0}: Error finding container bb2eaf51ed39abb2048c19a31a7c222efa081361f95d07904d4f1bc7ac967009: Status 404 returned error can't find the container with id bb2eaf51ed39abb2048c19a31a7c222efa081361f95d07904d4f1bc7ac967009
Nov 02 15:16:00.085781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:00.085759    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"]
Nov 02 15:16:00.136212 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:00.136184    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" event={"ID":"4da5799a-02da-4d12-98ae-7eb568707007","Type":"ContainerStarted","Data":"bb2eaf51ed39abb2048c19a31a7c222efa081361f95d07904d4f1bc7ac967009"}
Nov 02 15:16:01.144825 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:01.144790    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" event={"ID":"4da5799a-02da-4d12-98ae-7eb568707007","Type":"ContainerStarted","Data":"a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0"}
Nov 02 15:16:01.265570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:01.265515    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podStartSLOduration=2.2655005089999998 podStartE2EDuration="2.265500509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:01.261932681 +0000 UTC m=+1168.854549481" watchObservedRunningTime="2026-11-02 15:16:01.265500509 +0000 UTC m=+1168.858117308"
Nov 02 15:16:01.698783 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:01.698750    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:16:03.054166 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.054118    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"]
Nov 02 15:16:03.058526 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.058501    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:03.119076 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.119043    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1a9bba9-8xmmc\" (UID: \"b3329707-e6d4-45b2-a2ad-df6384d1c895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:03.119259 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.119206    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1a9bba9-8xmmc\" (UID: \"b3329707-e6d4-45b2-a2ad-df6384d1c895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:03.119362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.119335    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cv5k\" (UniqueName: \"kubernetes.io/projected/b3329707-e6d4-45b2-a2ad-df6384d1c895-kube-api-access-2cv5k\") pod \"maas-api-revoke-sub-e1a9bba9-8xmmc\" (UID: \"b3329707-e6d4-45b2-a2ad-df6384d1c895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:03.120080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.120062    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:16:03.120147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.120069    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:16:03.120147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.120092    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:16:03.169841 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.169802    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:16:03.174916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.174886    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"]
Nov 02 15:16:03.220545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.220500    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cv5k\" (UniqueName: \"kubernetes.io/projected/b3329707-e6d4-45b2-a2ad-df6384d1c895-kube-api-access-2cv5k\") pod \"maas-api-revoke-sub-e1a9bba9-8xmmc\" (UID: \"b3329707-e6d4-45b2-a2ad-df6384d1c895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:03.220734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.220619    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1a9bba9-8xmmc\" (UID: \"b3329707-e6d4-45b2-a2ad-df6384d1c895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:03.220803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.220742    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1a9bba9-8xmmc\" (UID: \"b3329707-e6d4-45b2-a2ad-df6384d1c895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:03.221273 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.221247    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1a9bba9-8xmmc\" (UID: \"b3329707-e6d4-45b2-a2ad-df6384d1c895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:03.223292 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.223272    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1a9bba9-8xmmc\" (UID: \"b3329707-e6d4-45b2-a2ad-df6384d1c895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:03.281796 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.281756    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cv5k\" (UniqueName: \"kubernetes.io/projected/b3329707-e6d4-45b2-a2ad-df6384d1c895-kube-api-access-2cv5k\") pod \"maas-api-revoke-sub-e1a9bba9-8xmmc\" (UID: \"b3329707-e6d4-45b2-a2ad-df6384d1c895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:03.376404 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.376276    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:03.814460 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:16:03.814430    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3329707_e6d4_45b2_a2ad_df6384d1c895.slice/crio-d61ee697d02acebf7600ad037b88b4c7bfd77c9679dea6d613758ec745c09e30 WatchSource:0}: Error finding container d61ee697d02acebf7600ad037b88b4c7bfd77c9679dea6d613758ec745c09e30: Status 404 returned error can't find the container with id d61ee697d02acebf7600ad037b88b4c7bfd77c9679dea6d613758ec745c09e30
Nov 02 15:16:03.815878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:03.815848    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"]
Nov 02 15:16:04.185178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:04.185136    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc" event={"ID":"b3329707-e6d4-45b2-a2ad-df6384d1c895","Type":"ContainerStarted","Data":"d61ee697d02acebf7600ad037b88b4c7bfd77c9679dea6d613758ec745c09e30"}
Nov 02 15:16:05.196560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:05.196439    4891 generic.go:356] "Generic (PLEG): container finished" podID="b3329707-e6d4-45b2-a2ad-df6384d1c895" containerID="38e640b1ea86eb4e70062a35f95ded9adc98f1ad9d8ac0ee095da5c02a11791a" exitCode=0
Nov 02 15:16:05.196560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:05.196515    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc" event={"ID":"b3329707-e6d4-45b2-a2ad-df6384d1c895","Type":"ContainerDied","Data":"38e640b1ea86eb4e70062a35f95ded9adc98f1ad9d8ac0ee095da5c02a11791a"}
Nov 02 15:16:06.358138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:06.358115    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:06.475076 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:06.474979    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-xks-ca\") pod \"b3329707-e6d4-45b2-a2ad-df6384d1c895\" (UID: \"b3329707-e6d4-45b2-a2ad-df6384d1c895\") "
Nov 02 15:16:06.475076 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:06.475069    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-openshift-ca\") pod \"b3329707-e6d4-45b2-a2ad-df6384d1c895\" (UID: \"b3329707-e6d4-45b2-a2ad-df6384d1c895\") "
Nov 02 15:16:06.475347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:06.475133    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3329707-e6d4-45b2-a2ad-df6384d1c895-kube-api-access-2cv5k\" (UniqueName: \"kubernetes.io/projected/b3329707-e6d4-45b2-a2ad-df6384d1c895-kube-api-access-2cv5k\") pod \"b3329707-e6d4-45b2-a2ad-df6384d1c895\" (UID: \"b3329707-e6d4-45b2-a2ad-df6384d1c895\") "
Nov 02 15:16:06.475691 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:06.475651    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-openshift-ca" pod "b3329707-e6d4-45b2-a2ad-df6384d1c895" (UID: "b3329707-e6d4-45b2-a2ad-df6384d1c895"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:06.475838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:06.475785    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:06.478430 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:06.478385    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3329707-e6d4-45b2-a2ad-df6384d1c895-kube-api-access-2cv5k" pod "b3329707-e6d4-45b2-a2ad-df6384d1c895" (UID: "b3329707-e6d4-45b2-a2ad-df6384d1c895"). InnerVolumeSpecName "kube-api-access-2cv5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:16:06.481780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:06.481724    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-xks-ca" pod "b3329707-e6d4-45b2-a2ad-df6384d1c895" (UID: "b3329707-e6d4-45b2-a2ad-df6384d1c895"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:16:06.576918 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:06.576882    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3329707-e6d4-45b2-a2ad-df6384d1c895-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:06.576918 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:06.576917    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cv5k\" (UniqueName: \"kubernetes.io/projected/b3329707-e6d4-45b2-a2ad-df6384d1c895-kube-api-access-2cv5k\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:07.212780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:07.212742    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc" event={"ID":"b3329707-e6d4-45b2-a2ad-df6384d1c895","Type":"ContainerDied","Data":"d61ee697d02acebf7600ad037b88b4c7bfd77c9679dea6d613758ec745c09e30"}
Nov 02 15:16:07.212954 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:07.212803    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d61ee697d02acebf7600ad037b88b4c7bfd77c9679dea6d613758ec745c09e30"
Nov 02 15:16:07.212954 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:07.212760    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"
Nov 02 15:16:09.698340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:09.698284    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:16:09.710829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:09.710799    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:16:10.256285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:10.256255    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:16:14.274847 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:16:14.274800    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:16:19.966445 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:19.966399    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"]
Nov 02 15:16:19.966958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:19.966898    4891 cpu_manager.go:414] "containerMap: removing container" podUID="b3329707-e6d4-45b2-a2ad-df6384d1c895" containerName="revoke-keys"
Nov 02 15:16:19.966958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:19.966909    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3329707-e6d4-45b2-a2ad-df6384d1c895" containerName="revoke-keys"
Nov 02 15:16:19.967049 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:19.967003    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3329707-e6d4-45b2-a2ad-df6384d1c895" containerName="revoke-keys"
Nov 02 15:16:19.970545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:19.970520    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.042603 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.042565    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.042791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.042685    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcd7c\" (UniqueName: \"kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-kube-api-access-bcd7c\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.042791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.042757    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.042791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.042784    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.042930 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.042820    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.042930 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.042900    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.043004 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.042935    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.043004 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.042957    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.043004 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.042977    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.054844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.054807    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"]
Nov 02 15:16:20.076897 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.076864    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786ebfb1-466pn"]
Nov 02 15:16:20.144392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.144341    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.144583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.144404    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.144583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.144423    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.144583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.144461    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.144583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.144481    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.144583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.144506    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.144583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.144529    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.144914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.144633    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.144914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.144737    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcd7c\" (UniqueName: \"kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-kube-api-access-bcd7c\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.144914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.144901    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.145081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.144988    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.145205 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.145177    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.145300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.145265    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.145405 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.145379    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.147385 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.147357    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.147643 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.147621    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.174847 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.174815    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"]
Nov 02 15:16:20.205907 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.205865    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.206089 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.205984    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcd7c\" (UniqueName: \"kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-kube-api-access-bcd7c\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.284412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.284373    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:20.345039 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.345007    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"]
Nov 02 15:16:20.350615 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.350585    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.439789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.439740    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"]
Nov 02 15:16:20.449208 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.449179    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.449419 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.449228    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d46n\" (UniqueName: \"kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-kube-api-access-6d46n\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.449419 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.449262    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.449419 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.449396    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.449605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.449446    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.449605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.449475    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.449605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.449519    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83865724-7220-4274-9262-f2b09ff911a1-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.449605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.449550    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.449605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.449583    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83865724-7220-4274-9262-f2b09ff911a1-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.524602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.524564    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7cacf64-9854-423d-89ae-6bdae3d68681" path="/var/lib/kubelet/pods/f7cacf64-9854-423d-89ae-6bdae3d68681/volumes"
Nov 02 15:16:20.550942 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.550866    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.550942 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.550905    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.551162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.550955    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83865724-7220-4274-9262-f2b09ff911a1-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.551162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.550980    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.551162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.551007    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83865724-7220-4274-9262-f2b09ff911a1-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.551162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.551088    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.551162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.551129    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d46n\" (UniqueName: \"kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-kube-api-access-6d46n\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.551462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.551166    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.551462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.551305    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.551462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.551353    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.551462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.551417    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.551700 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.551664    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.551700 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.551651    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.552076 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.552053    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83865724-7220-4274-9262-f2b09ff911a1-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.553130 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.553108    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.553417 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.553401    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83865724-7220-4274-9262-f2b09ff911a1-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.618961 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.618928    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d46n\" (UniqueName: \"kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-kube-api-access-6d46n\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.619280 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.619261    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.678600 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.678549    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:20.678968 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:16:20.678940    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d1d6d1_77ba_4dca_9b35_dbeb471cf2e5.slice/crio-fc3005481bcbc1f8ee21c8ca4d10394ed60261fac10998dc61fe940db3a37655 WatchSource:0}: Error finding container fc3005481bcbc1f8ee21c8ca4d10394ed60261fac10998dc61fe940db3a37655: Status 404 returned error can't find the container with id fc3005481bcbc1f8ee21c8ca4d10394ed60261fac10998dc61fe940db3a37655
Nov 02 15:16:20.681052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.681025    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"]
Nov 02 15:16:20.691341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.683195    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:20.691341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.688523    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:20.691341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:20.688615    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:21.078735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.078707    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"]
Nov 02 15:16:21.080470 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:16:21.080445    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83865724_7220_4274_9262_f2b09ff911a1.slice/crio-e96f92e2e9aa488bd9f22a37f4eb8695d988afb9bb498e9ebf85a98c7b79f0a8 WatchSource:0}: Error finding container e96f92e2e9aa488bd9f22a37f4eb8695d988afb9bb498e9ebf85a98c7b79f0a8: Status 404 returned error can't find the container with id e96f92e2e9aa488bd9f22a37f4eb8695d988afb9bb498e9ebf85a98c7b79f0a8
Nov 02 15:16:21.083700 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.083661    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:21.083812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.083732    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:21.083812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.083760    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:21.164568 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.164459    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"]
Nov 02 15:16:21.169487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.169464    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:21.222232 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.222202    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:16:21.222232 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.222217    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:16:21.222554 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.222273    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:16:21.222554 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.222339    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:16:21.227630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.227526    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"]
Nov 02 15:16:21.266138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.266061    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a8c556b-ph47s\" (UID: \"094d6dba-849b-4faa-b1f7-8725979638d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:21.266138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.266102    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a8c556b-ph47s\" (UID: \"094d6dba-849b-4faa-b1f7-8725979638d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:21.267758 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.266281    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkflv\" (UniqueName: \"kubernetes.io/projected/094d6dba-849b-4faa-b1f7-8725979638d1-kube-api-access-jkflv\") pod \"maas-api-revoke-sub-1a8c556b-ph47s\" (UID: \"094d6dba-849b-4faa-b1f7-8725979638d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:21.333053 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.332959    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" event={"ID":"83865724-7220-4274-9262-f2b09ff911a1","Type":"ContainerStarted","Data":"4e140dc090368b4e27dac35760e43959872834121491b71e38ce9a871a86ccf9"}
Nov 02 15:16:21.333053 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.333022    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" event={"ID":"83865724-7220-4274-9262-f2b09ff911a1","Type":"ContainerStarted","Data":"e96f92e2e9aa488bd9f22a37f4eb8695d988afb9bb498e9ebf85a98c7b79f0a8"}
Nov 02 15:16:21.334800 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.334773    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" event={"ID":"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5","Type":"ContainerStarted","Data":"126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996"}
Nov 02 15:16:21.334921 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.334825    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" event={"ID":"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5","Type":"ContainerStarted","Data":"fc3005481bcbc1f8ee21c8ca4d10394ed60261fac10998dc61fe940db3a37655"}
Nov 02 15:16:21.367818 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.367772    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a8c556b-ph47s\" (UID: \"094d6dba-849b-4faa-b1f7-8725979638d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:21.368009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.367836    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a8c556b-ph47s\" (UID: \"094d6dba-849b-4faa-b1f7-8725979638d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:21.368009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.367955    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkflv\" (UniqueName: \"kubernetes.io/projected/094d6dba-849b-4faa-b1f7-8725979638d1-kube-api-access-jkflv\") pod \"maas-api-revoke-sub-1a8c556b-ph47s\" (UID: \"094d6dba-849b-4faa-b1f7-8725979638d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:21.368833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.368808    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a8c556b-ph47s\" (UID: \"094d6dba-849b-4faa-b1f7-8725979638d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:21.370676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.370612    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a8c556b-ph47s\" (UID: \"094d6dba-849b-4faa-b1f7-8725979638d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:21.429571 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.429531    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkflv\" (UniqueName: \"kubernetes.io/projected/094d6dba-849b-4faa-b1f7-8725979638d1-kube-api-access-jkflv\") pod \"maas-api-revoke-sub-1a8c556b-ph47s\" (UID: \"094d6dba-849b-4faa-b1f7-8725979638d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:21.466906 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.460821    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" podStartSLOduration=1.460799409 podStartE2EDuration="1.460799409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:21.455253707 +0000 UTC m=+1189.047870508" watchObservedRunningTime="2026-11-02 15:16:21.460799409 +0000 UTC m=+1189.053416212"
Nov 02 15:16:21.488564 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.488524    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:21.678645 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.678612    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:21.769335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.765123    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" podStartSLOduration=2.765102928 podStartE2EDuration="2.765102928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:21.584917725 +0000 UTC m=+1189.177534525" watchObservedRunningTime="2026-11-02 15:16:21.765102928 +0000 UTC m=+1189.357719727"
Nov 02 15:16:21.770870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.770832    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"]
Nov 02 15:16:21.781943 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.781915    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:21.843770 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.843620    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"]
Nov 02 15:16:21.844240 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.844214    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-c18e79-openshift-default-dockercfg-2fs8l\""
Nov 02 15:16:21.888560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.887875    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:21.888560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.887961    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwlfz\" (UniqueName: \"kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-kube-api-access-kwlfz\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:21.888560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.888012    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:21.888560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.888038    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:21.888560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.888077    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:21.888560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.888118    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c325c48e-6940-4af7-b486-488df4754813-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:21.888560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.888160    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:21.888560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.888249    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:21.888560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.888276    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c325c48e-6940-4af7-b486-488df4754813-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:21.923593 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.922579    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"]
Nov 02 15:16:21.999945 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:21.999788    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.009622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.009595    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.009824 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.009805    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.015427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.015397    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c325c48e-6940-4af7-b486-488df4754813-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.015618 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.015599    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.015865 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.015840    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.016002 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.015984    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c325c48e-6940-4af7-b486-488df4754813-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.016252 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.016236    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.020595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.020563    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwlfz\" (UniqueName: \"kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-kube-api-access-kwlfz\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.020883 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.018040    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.020970 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.008856    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.020970 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.018266    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.020970 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.019552    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.020970 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.017091    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.021891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.021858    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c325c48e-6940-4af7-b486-488df4754813-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.021982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.021909    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c325c48e-6940-4af7-b486-488df4754813-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.158384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.156637    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwlfz\" (UniqueName: \"kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-kube-api-access-kwlfz\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.158384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.157090    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.286576 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.286458    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:22.294290 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.293452    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 15:16:22.294290 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.293513    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 15:16:22.345367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.344814    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s" event={"ID":"094d6dba-849b-4faa-b1f7-8725979638d1","Type":"ContainerStarted","Data":"53e0a96f23077750fe4a3ede5189cba7e13276d4f62eab6ab9704daabfa91347"}
Nov 02 15:16:22.401730 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.401690    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:22.693875 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.693784    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:16:22.694132 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.694104    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:16:22.914184 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.914156    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"]
Nov 02 15:16:22.918051 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:16:22.917546    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc325c48e_6940_4af7_b486_488df4754813.slice/crio-14ac8edb72caae06a5073ea79d8b4e8c0d4f7b95146279471ac9204ae9fe7102 WatchSource:0}: Error finding container 14ac8edb72caae06a5073ea79d8b4e8c0d4f7b95146279471ac9204ae9fe7102: Status 404 returned error can't find the container with id 14ac8edb72caae06a5073ea79d8b4e8c0d4f7b95146279471ac9204ae9fe7102
Nov 02 15:16:22.921600 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.921558    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:22.922210 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.922158    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:22.922345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:22.922299    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:23.291178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:23.290713    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 15:16:23.291178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:23.290776    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 15:16:23.385335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:23.376879    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" event={"ID":"c325c48e-6940-4af7-b486-488df4754813","Type":"ContainerStarted","Data":"f321cd99b7e07d9671844b0a7eafc9242c5858f88bc57493a02d3c1300e5d6cc"}
Nov 02 15:16:23.385335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:23.377019    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" event={"ID":"c325c48e-6940-4af7-b486-488df4754813","Type":"ContainerStarted","Data":"14ac8edb72caae06a5073ea79d8b4e8c0d4f7b95146279471ac9204ae9fe7102"}
Nov 02 15:16:23.385335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:23.379614    4891 generic.go:356] "Generic (PLEG): container finished" podID="094d6dba-849b-4faa-b1f7-8725979638d1" containerID="a92d2eeb613201bdab589fdf9a5c8d5ed1d7f3a121820b9abfb06b0f8db5a003" exitCode=0
Nov 02 15:16:23.385335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:23.380364    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s" event={"ID":"094d6dba-849b-4faa-b1f7-8725979638d1","Type":"ContainerDied","Data":"a92d2eeb613201bdab589fdf9a5c8d5ed1d7f3a121820b9abfb06b0f8db5a003"}
Nov 02 15:16:23.402478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:23.402402    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:23.628166 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:23.628101    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" podStartSLOduration=2.628081152 podStartE2EDuration="2.628081152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:23.510935538 +0000 UTC m=+1191.103552337" watchObservedRunningTime="2026-11-02 15:16:23.628081152 +0000 UTC m=+1191.220697953"
Nov 02 15:16:23.679763 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:23.679719    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:16:23.680945 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:23.680903    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:16:24.285741 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:24.285703    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 15:16:24.286088 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:24.286056    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 15:16:24.414508 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:24.408705    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:16:24.414508 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:24.408773    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" podUID="c325c48e-6940-4af7-b486-488df4754813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:16:24.738333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:24.710384    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:16:24.738333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:24.730621    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:16:25.004337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.003675    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:25.129338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.127587    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-xks-ca\") pod \"094d6dba-849b-4faa-b1f7-8725979638d1\" (UID: \"094d6dba-849b-4faa-b1f7-8725979638d1\") "
Nov 02 15:16:25.129338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.127798    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-openshift-ca\") pod \"094d6dba-849b-4faa-b1f7-8725979638d1\" (UID: \"094d6dba-849b-4faa-b1f7-8725979638d1\") "
Nov 02 15:16:25.129338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.127955    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094d6dba-849b-4faa-b1f7-8725979638d1-kube-api-access-jkflv\" (UniqueName: \"kubernetes.io/projected/094d6dba-849b-4faa-b1f7-8725979638d1-kube-api-access-jkflv\") pod \"094d6dba-849b-4faa-b1f7-8725979638d1\" (UID: \"094d6dba-849b-4faa-b1f7-8725979638d1\") "
Nov 02 15:16:25.133338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.129843    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-openshift-ca" pod "094d6dba-849b-4faa-b1f7-8725979638d1" (UID: "094d6dba-849b-4faa-b1f7-8725979638d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:25.244339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.242447    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-xks-ca" pod "094d6dba-849b-4faa-b1f7-8725979638d1" (UID: "094d6dba-849b-4faa-b1f7-8725979638d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:16:25.244339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.243755    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:25.244339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.243783    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094d6dba-849b-4faa-b1f7-8725979638d1-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:25.270330 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.262245    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094d6dba-849b-4faa-b1f7-8725979638d1-kube-api-access-jkflv" pod "094d6dba-849b-4faa-b1f7-8725979638d1" (UID: "094d6dba-849b-4faa-b1f7-8725979638d1"). InnerVolumeSpecName "kube-api-access-jkflv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:16:25.288370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.286744    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 15:16:25.288370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.286803    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 15:16:25.348338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.344805    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkflv\" (UniqueName: \"kubernetes.io/projected/094d6dba-849b-4faa-b1f7-8725979638d1-kube-api-access-jkflv\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:25.415334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.414442    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:16:25.415334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.414507    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" podUID="c325c48e-6940-4af7-b486-488df4754813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:16:25.492841 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.492750    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"
Nov 02 15:16:25.506986 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.493075    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s" event={"ID":"094d6dba-849b-4faa-b1f7-8725979638d1","Type":"ContainerDied","Data":"53e0a96f23077750fe4a3ede5189cba7e13276d4f62eab6ab9704daabfa91347"}
Nov 02 15:16:25.506986 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.504635    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53e0a96f23077750fe4a3ede5189cba7e13276d4f62eab6ab9704daabfa91347"
Nov 02 15:16:25.701356 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.700456    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:16:25.701356 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:25.700543    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:16:26.286337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:26.285638    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 15:16:26.286337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:26.285706    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 15:16:26.403340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:26.402676    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:16:26.403340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:26.402737    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" podUID="c325c48e-6940-4af7-b486-488df4754813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:16:26.681455 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:26.680733    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:16:26.681455 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:26.680812    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:16:27.289332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:27.288916    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 15:16:27.289332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:27.289002    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 15:16:27.323350 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:27.311451    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:16:27.410154 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:27.410108    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:16:27.410379 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:27.410181    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" podUID="c325c48e-6940-4af7-b486-488df4754813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:16:27.664386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:27.661443    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:16:27.682729 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:27.682694    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:16:27.683638 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:27.683611    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:16:27.700413 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:27.700374    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:16:27.906331 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:27.904556    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:16:28.286334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:28.285708    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 15:16:28.286334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:28.285768    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 15:16:28.404330 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:28.404244    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:16:28.404535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:28.404387    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" podUID="c325c48e-6940-4af7-b486-488df4754813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:16:28.693345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:28.693233    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 15:16:28.693926 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:28.693870    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 15:16:29.402889 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:29.402849    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:16:29.403094 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:29.402897    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" podUID="c325c48e-6940-4af7-b486-488df4754813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:16:29.551072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:29.551041    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:29.552778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:29.552757    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:29.553973 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:29.553952    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:16:30.402789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:30.402745    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:16:30.403181 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:30.402818    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" podUID="c325c48e-6940-4af7-b486-488df4754813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 15:16:30.732539 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:30.732456    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:16:30.732539 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:30.732528    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:16:31.053449 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:31.053406    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"]
Nov 02 15:16:31.058068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:31.058031    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df4cae22-wrl77"]
Nov 02 15:16:31.309732 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:31.309627    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:31.310186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:31.310163    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:31.310971 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:31.310953    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:16:31.455961 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:31.455927    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:31.456446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:31.456424    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:31.457176 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:31.457159    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:16:32.524186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:32.524157    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707379e5-d3ea-4f57-89fa-27e0b659290f" path="/var/lib/kubelet/pods/707379e5-d3ea-4f57-89fa-27e0b659290f/volumes"
Nov 02 15:16:32.982376 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:32.982273    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:16:32.982658 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:32.982637    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:16:32.983511 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:32.983484    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:16:32.983664 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:32.983638    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:16:32.998946 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:32.998919    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:16:32.999161 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:32.999142    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:16:35.383975 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.383941    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"]
Nov 02 15:16:35.384500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.384484    4891 cpu_manager.go:414] "containerMap: removing container" podUID="094d6dba-849b-4faa-b1f7-8725979638d1" containerName="revoke-keys"
Nov 02 15:16:35.384543 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.384501    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094d6dba-849b-4faa-b1f7-8725979638d1" containerName="revoke-keys"
Nov 02 15:16:35.384634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.384625    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="094d6dba-849b-4faa-b1f7-8725979638d1" containerName="revoke-keys"
Nov 02 15:16:35.389374 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.389356    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:35.433663 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.433625    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmkmp\" (UniqueName: \"kubernetes.io/projected/956b7542-522f-4792-a4b8-8ec900796bab-kube-api-access-qmkmp\") pod \"maas-api-revoke-sub-9bb05927-7hrq4\" (UID: \"956b7542-522f-4792-a4b8-8ec900796bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:35.433829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.433696    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bb05927-7hrq4\" (UID: \"956b7542-522f-4792-a4b8-8ec900796bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:35.433889 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.433863    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bb05927-7hrq4\" (UID: \"956b7542-522f-4792-a4b8-8ec900796bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:35.445917 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.445882    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"]
Nov 02 15:16:35.489840 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.489802    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:16:35.490005 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.489836    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:16:35.490005 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.489855    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:16:35.490943 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.490917    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:16:35.535271 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.535244    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkmp\" (UniqueName: \"kubernetes.io/projected/956b7542-522f-4792-a4b8-8ec900796bab-kube-api-access-qmkmp\") pod \"maas-api-revoke-sub-9bb05927-7hrq4\" (UID: \"956b7542-522f-4792-a4b8-8ec900796bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:35.535427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.535344    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bb05927-7hrq4\" (UID: \"956b7542-522f-4792-a4b8-8ec900796bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:35.535683 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.535647    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bb05927-7hrq4\" (UID: \"956b7542-522f-4792-a4b8-8ec900796bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:35.536156 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.536136    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bb05927-7hrq4\" (UID: \"956b7542-522f-4792-a4b8-8ec900796bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:35.537932 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.537911    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bb05927-7hrq4\" (UID: \"956b7542-522f-4792-a4b8-8ec900796bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:35.595846 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.595816    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkmp\" (UniqueName: \"kubernetes.io/projected/956b7542-522f-4792-a4b8-8ec900796bab-kube-api-access-qmkmp\") pod \"maas-api-revoke-sub-9bb05927-7hrq4\" (UID: \"956b7542-522f-4792-a4b8-8ec900796bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:35.704473 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:35.704388    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:36.075179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.075137    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"]
Nov 02 15:16:36.128086 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.121356    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"]
Nov 02 15:16:36.128086 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.127876    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6fdd986-5tddp"]
Nov 02 15:16:36.523279 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.523248    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c9fe1b-828d-4328-a553-ccdf0498673a" path="/var/lib/kubelet/pods/26c9fe1b-828d-4328-a553-ccdf0498673a/volumes"
Nov 02 15:16:36.705791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.705706    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4" event={"ID":"956b7542-522f-4792-a4b8-8ec900796bab","Type":"ContainerStarted","Data":"7ecef08172e9f12eadd47646e5166d4956d36bfe49905aa4c60ed729ee36d07a"}
Nov 02 15:16:36.705791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.705760    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4" event={"ID":"956b7542-522f-4792-a4b8-8ec900796bab","Type":"ContainerStarted","Data":"ed58215ff2e39b14982ae2eb7677ec6b3633249aaef2feed3241793c2aa8542a"}
Nov 02 15:16:36.759905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.759866    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"]
Nov 02 15:16:36.764078 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.764054    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:36.822491 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.822438    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"]
Nov 02 15:16:36.823508 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.823457    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4" podStartSLOduration=1.823438329 podStartE2EDuration="1.823438329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:36.82039254 +0000 UTC m=+1204.413009339" watchObservedRunningTime="2026-11-02 15:16:36.823438329 +0000 UTC m=+1204.416055129"
Nov 02 15:16:36.855289 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.855245    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pm8n\" (UniqueName: \"kubernetes.io/projected/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-kube-api-access-6pm8n\") pod \"maas-api-revoke-sub-06b85ca0-z9l47\" (UID: \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:36.855471 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.855338    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06b85ca0-z9l47\" (UID: \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:36.855570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.855539    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06b85ca0-z9l47\" (UID: \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:36.957100 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.957020    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06b85ca0-z9l47\" (UID: \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:36.957236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.957171    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pm8n\" (UniqueName: \"kubernetes.io/projected/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-kube-api-access-6pm8n\") pod \"maas-api-revoke-sub-06b85ca0-z9l47\" (UID: \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:36.957368 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.957350    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06b85ca0-z9l47\" (UID: \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:36.957955 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.957937    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06b85ca0-z9l47\" (UID: \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:36.959438 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:36.959423    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06b85ca0-z9l47\" (UID: \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:37.018042 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:37.018015    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pm8n\" (UniqueName: \"kubernetes.io/projected/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-kube-api-access-6pm8n\") pod \"maas-api-revoke-sub-06b85ca0-z9l47\" (UID: \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:37.078019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:37.077992    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:37.461406 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:37.461366    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"]
Nov 02 15:16:37.462052 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:16:37.462015    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57d1e3a_2dfb_4067_9b3b_e6d8b1bc6d7e.slice/crio-aa583e39ff7e4080ccc54014421c5c85f56b8cf271fd1ba5b3494903439b3cf0 WatchSource:0}: Error finding container aa583e39ff7e4080ccc54014421c5c85f56b8cf271fd1ba5b3494903439b3cf0: Status 404 returned error can't find the container with id aa583e39ff7e4080ccc54014421c5c85f56b8cf271fd1ba5b3494903439b3cf0
Nov 02 15:16:37.714559 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:37.714474    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47" event={"ID":"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e","Type":"ContainerStarted","Data":"aa583e39ff7e4080ccc54014421c5c85f56b8cf271fd1ba5b3494903439b3cf0"}
Nov 02 15:16:37.716137 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:37.716108    4891 generic.go:356] "Generic (PLEG): container finished" podID="956b7542-522f-4792-a4b8-8ec900796bab" containerID="7ecef08172e9f12eadd47646e5166d4956d36bfe49905aa4c60ed729ee36d07a" exitCode=0
Nov 02 15:16:37.716253 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:37.716199    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4" event={"ID":"956b7542-522f-4792-a4b8-8ec900796bab","Type":"ContainerDied","Data":"7ecef08172e9f12eadd47646e5166d4956d36bfe49905aa4c60ed729ee36d07a"}
Nov 02 15:16:38.725391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:38.725276    4891 generic.go:356] "Generic (PLEG): container finished" podID="c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e" containerID="e05c47c73a5587a5e51b1b9e7257583c7c0e7727f6dc774c0a6c70c891e18e7b" exitCode=0
Nov 02 15:16:38.725391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:38.725360    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47" event={"ID":"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e","Type":"ContainerDied","Data":"e05c47c73a5587a5e51b1b9e7257583c7c0e7727f6dc774c0a6c70c891e18e7b"}
Nov 02 15:16:38.885431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:38.885405    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:38.981660 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:38.981588    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956b7542-522f-4792-a4b8-8ec900796bab-kube-api-access-qmkmp\" (UniqueName: \"kubernetes.io/projected/956b7542-522f-4792-a4b8-8ec900796bab-kube-api-access-qmkmp\") pod \"956b7542-522f-4792-a4b8-8ec900796bab\" (UID: \"956b7542-522f-4792-a4b8-8ec900796bab\") "
Nov 02 15:16:38.981793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:38.981720    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-xks-ca\") pod \"956b7542-522f-4792-a4b8-8ec900796bab\" (UID: \"956b7542-522f-4792-a4b8-8ec900796bab\") "
Nov 02 15:16:38.981856 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:38.981845    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-openshift-ca\") pod \"956b7542-522f-4792-a4b8-8ec900796bab\" (UID: \"956b7542-522f-4792-a4b8-8ec900796bab\") "
Nov 02 15:16:38.982251 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:38.982217    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-openshift-ca" pod "956b7542-522f-4792-a4b8-8ec900796bab" (UID: "956b7542-522f-4792-a4b8-8ec900796bab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:38.983781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:38.983753    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956b7542-522f-4792-a4b8-8ec900796bab-kube-api-access-qmkmp" pod "956b7542-522f-4792-a4b8-8ec900796bab" (UID: "956b7542-522f-4792-a4b8-8ec900796bab"). InnerVolumeSpecName "kube-api-access-qmkmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:16:38.983781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:38.983770    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-xks-ca" pod "956b7542-522f-4792-a4b8-8ec900796bab" (UID: "956b7542-522f-4792-a4b8-8ec900796bab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:16:39.083379 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.083341    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:39.083379 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.083377    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/956b7542-522f-4792-a4b8-8ec900796bab-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:39.083594 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.083391    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmkmp\" (UniqueName: \"kubernetes.io/projected/956b7542-522f-4792-a4b8-8ec900796bab-kube-api-access-qmkmp\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:39.734712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.734682    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"
Nov 02 15:16:39.735176 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.734719    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4" event={"ID":"956b7542-522f-4792-a4b8-8ec900796bab","Type":"ContainerDied","Data":"ed58215ff2e39b14982ae2eb7677ec6b3633249aaef2feed3241793c2aa8542a"}
Nov 02 15:16:39.735176 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.734769    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed58215ff2e39b14982ae2eb7677ec6b3633249aaef2feed3241793c2aa8542a"
Nov 02 15:16:39.890974 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.890947    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:39.995232 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.995144    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-openshift-ca\") pod \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\" (UID: \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\") "
Nov 02 15:16:39.995438 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.995347    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-kube-api-access-6pm8n\" (UniqueName: \"kubernetes.io/projected/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-kube-api-access-6pm8n\") pod \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\" (UID: \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\") "
Nov 02 15:16:39.995538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.995509    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-openshift-ca" pod "c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e" (UID: "c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:39.995614 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.995517    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-xks-ca\") pod \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\" (UID: \"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e\") "
Nov 02 15:16:39.995957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.995940    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:39.997333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.997297    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-kube-api-access-6pm8n" pod "c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e" (UID: "c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e"). InnerVolumeSpecName "kube-api-access-6pm8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:16:39.997442 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:39.997417    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-xks-ca" pod "c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e" (UID: "c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:16:40.097424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:40.097394    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pm8n\" (UniqueName: \"kubernetes.io/projected/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-kube-api-access-6pm8n\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:40.097424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:40.097423    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:40.753037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:40.753008    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"
Nov 02 15:16:40.753482 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:40.753049    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47" event={"ID":"c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e","Type":"ContainerDied","Data":"aa583e39ff7e4080ccc54014421c5c85f56b8cf271fd1ba5b3494903439b3cf0"}
Nov 02 15:16:40.753482 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:40.753141    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa583e39ff7e4080ccc54014421c5c85f56b8cf271fd1ba5b3494903439b3cf0"
Nov 02 15:16:44.274557 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:16:44.274480    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:16:47.077592 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:47.077553    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"]
Nov 02 15:16:47.128381 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:47.128340    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf5104d4-tnmq9"]
Nov 02 15:16:48.530926 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:48.530892    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f75a66-4ca5-4463-836f-d1120ae748db" path="/var/lib/kubelet/pods/51f75a66-4ca5-4463-836f-d1120ae748db/volumes"
Nov 02 15:16:50.923202 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:50.923166    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"]
Nov 02 15:16:50.923739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:50.923723    4891 cpu_manager.go:414] "containerMap: removing container" podUID="c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e" containerName="revoke-keys"
Nov 02 15:16:50.923788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:50.923740    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e" containerName="revoke-keys"
Nov 02 15:16:50.923788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:50.923751    4891 cpu_manager.go:414] "containerMap: removing container" podUID="956b7542-522f-4792-a4b8-8ec900796bab" containerName="revoke-keys"
Nov 02 15:16:50.923788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:50.923756    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956b7542-522f-4792-a4b8-8ec900796bab" containerName="revoke-keys"
Nov 02 15:16:50.923889 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:50.923856    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e" containerName="revoke-keys"
Nov 02 15:16:50.923889 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:50.923866    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="956b7542-522f-4792-a4b8-8ec900796bab" containerName="revoke-keys"
Nov 02 15:16:50.927355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:50.927329    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:50.986059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:50.986027    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"]
Nov 02 15:16:51.029326 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.029280    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hlh9\" (UniqueName: \"kubernetes.io/projected/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-kube-api-access-7hlh9\") pod \"maas-api-revoke-sub-32de075f-7zv58\" (UID: \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:51.029518 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.029441    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32de075f-7zv58\" (UID: \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:51.029605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.029580    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32de075f-7zv58\" (UID: \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:51.029917 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.029897    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:16:51.029917 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.029904    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:16:51.030074 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.029966    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:16:51.031366 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.031352    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:16:51.130660 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.130620    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32de075f-7zv58\" (UID: \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:51.130857 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.130736    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32de075f-7zv58\" (UID: \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:51.130857 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.130852    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hlh9\" (UniqueName: \"kubernetes.io/projected/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-kube-api-access-7hlh9\") pod \"maas-api-revoke-sub-32de075f-7zv58\" (UID: \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:51.131543 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.131522    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32de075f-7zv58\" (UID: \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:51.133104 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.133087    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32de075f-7zv58\" (UID: \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:51.193996 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.193928    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hlh9\" (UniqueName: \"kubernetes.io/projected/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-kube-api-access-7hlh9\") pod \"maas-api-revoke-sub-32de075f-7zv58\" (UID: \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:51.241268 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.241234    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:51.552493 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.552456    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"]
Nov 02 15:16:51.556666 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.556646    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.637364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.637329    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.637364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.637364    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2be07f0-3816-40ea-96f3-3eced30935e9-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.637602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.637391    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.637602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.637467    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.637602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.637544    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.637602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.637578    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.637777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.637620    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.637777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.637716    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.637777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.637751    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhrf\" (UniqueName: \"kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-kube-api-access-klhrf\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.672306 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.672281    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"]
Nov 02 15:16:51.675196 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:16:51.675166    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fd0ace_8fc1_42d8_bbf4_2123ae0ff35a.slice/crio-c57fa25a384ce4a533668be4b07d75a04640b58a18c64ab0863940b0df5b34c2 WatchSource:0}: Error finding container c57fa25a384ce4a533668be4b07d75a04640b58a18c64ab0863940b0df5b34c2: Status 404 returned error can't find the container with id c57fa25a384ce4a533668be4b07d75a04640b58a18c64ab0863940b0df5b34c2
Nov 02 15:16:51.738910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.738873    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739044 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.738918    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2be07f0-3816-40ea-96f3-3eced30935e9-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739111 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.739036    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739111 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.739092    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.739154    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.739189    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739348 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.739251    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739415 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.739389    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739467 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.739443    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhrf\" (UniqueName: \"kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-kube-api-access-klhrf\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739467 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.739442    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739636 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.739526    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739636 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.739549    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739796 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.739774    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.739873 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.739851    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2be07f0-3816-40ea-96f3-3eced30935e9-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.741740 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.741717    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.741968 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.741949    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.769009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.768968    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"]
Nov 02 15:16:51.800390 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.800355    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.800557 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.800547    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhrf\" (UniqueName: \"kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-kube-api-access-klhrf\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:51.845592 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.845522    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58" event={"ID":"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a","Type":"ContainerStarted","Data":"c57fa25a384ce4a533668be4b07d75a04640b58a18c64ab0863940b0df5b34c2"}
Nov 02 15:16:51.870926 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:51.870901    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:52.316113 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:52.316075    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"]
Nov 02 15:16:52.317542 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:16:52.317500    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2be07f0_3816_40ea_96f3_3eced30935e9.slice/crio-2f53f8b3dba21e4f3648577e5c025e9d91666bfd7fbac66e985fe4b8e1ef8a55 WatchSource:0}: Error finding container 2f53f8b3dba21e4f3648577e5c025e9d91666bfd7fbac66e985fe4b8e1ef8a55: Status 404 returned error can't find the container with id 2f53f8b3dba21e4f3648577e5c025e9d91666bfd7fbac66e985fe4b8e1ef8a55
Nov 02 15:16:52.320160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:52.320123    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:52.320239 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:52.320194    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:52.320239 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:52.320222    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:52.856753 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:52.856717    4891 generic.go:356] "Generic (PLEG): container finished" podID="77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a" containerID="687da2b75e1df7344a210b9f28897a69edadad7abf5bd7566529d14d1f901e66" exitCode=0
Nov 02 15:16:52.856944 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:52.856804    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58" event={"ID":"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a","Type":"ContainerDied","Data":"687da2b75e1df7344a210b9f28897a69edadad7abf5bd7566529d14d1f901e66"}
Nov 02 15:16:52.858591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:52.858566    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" event={"ID":"d2be07f0-3816-40ea-96f3-3eced30935e9","Type":"ContainerStarted","Data":"14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05"}
Nov 02 15:16:52.858713 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:52.858606    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" event={"ID":"d2be07f0-3816-40ea-96f3-3eced30935e9","Type":"ContainerStarted","Data":"2f53f8b3dba21e4f3648577e5c025e9d91666bfd7fbac66e985fe4b8e1ef8a55"}
Nov 02 15:16:52.871174 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:52.871150    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:53.101593 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:53.101529    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" podStartSLOduration=2.1015105 podStartE2EDuration="2.1015105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:53.095935432 +0000 UTC m=+1220.688552231" watchObservedRunningTime="2026-11-02 15:16:53.1015105 +0000 UTC m=+1220.694127311"
Nov 02 15:16:53.873731 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:53.873698    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 15:16:53.874155 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:53.873747    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 15:16:54.071416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.071381    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:54.077835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.077814    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-kube-api-access-7hlh9\" (UniqueName: \"kubernetes.io/projected/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-kube-api-access-7hlh9\") pod \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\" (UID: \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\") "
Nov 02 15:16:54.077944 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.077885    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-openshift-ca\") pod \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\" (UID: \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\") "
Nov 02 15:16:54.078233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.078208    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-openshift-ca" pod "77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a" (UID: "77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:54.079997 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.079972    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-kube-api-access-7hlh9" pod "77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a" (UID: "77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a"). InnerVolumeSpecName "kube-api-access-7hlh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:16:54.179015 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.178933    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-xks-ca\") pod \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\" (UID: \"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a\") "
Nov 02 15:16:54.179241 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.179228    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hlh9\" (UniqueName: \"kubernetes.io/projected/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-kube-api-access-7hlh9\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:54.179288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.179244    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:54.180941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.180917    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-xks-ca" pod "77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a" (UID: "77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:16:54.280705 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.280662    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:16:54.872119 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.872030    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 15:16:54.872119 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.872084    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 15:16:54.883844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.883810    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"
Nov 02 15:16:54.883844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.883835    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58" event={"ID":"77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a","Type":"ContainerDied","Data":"c57fa25a384ce4a533668be4b07d75a04640b58a18c64ab0863940b0df5b34c2"}
Nov 02 15:16:54.884268 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:54.883884    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c57fa25a384ce4a533668be4b07d75a04640b58a18c64ab0863940b0df5b34c2"
Nov 02 15:16:55.872350 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:55.872274    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 15:16:55.872583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:55.872351    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 15:16:56.772423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.772387    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"]
Nov 02 15:16:56.772958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.772941    4891 cpu_manager.go:414] "containerMap: removing container" podUID="77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a" containerName="revoke-keys"
Nov 02 15:16:56.773020 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.772960    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a" containerName="revoke-keys"
Nov 02 15:16:56.773069 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.773058    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a" containerName="revoke-keys"
Nov 02 15:16:56.776260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.776238    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.808296 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.808265    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-dshm\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.808483 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.808302    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7s27\" (UniqueName: \"kubernetes.io/projected/ce7b021f-cc78-4f81-9160-dedfde5efa2e-kube-api-access-q7s27\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.808483 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.808343    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tls-certs\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.808483 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.808421    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-model-cache\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.808483 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.808439    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tmp-dir\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.808705 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.808543    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-home\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.835168 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.835131    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"]
Nov 02 15:16:56.872379 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.871705    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 15:16:56.872379 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.871764    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 15:16:56.878161 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.878130    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6dc69b\"/\"distinct-2-w4-6dc69b-kserve-self-signed-certs\""
Nov 02 15:16:56.910693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.909492    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tls-certs\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.910693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.909558    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-model-cache\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.910693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.909584    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tmp-dir\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.910693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.909666    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-home\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.910693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.909901    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-dshm\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.910693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.909934    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7s27\" (UniqueName: \"kubernetes.io/projected/ce7b021f-cc78-4f81-9160-dedfde5efa2e-kube-api-access-q7s27\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.911546 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.911521    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tmp-dir\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.911953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.911932    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-model-cache\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.912359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.912339    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-home\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.920815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.920406    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tls-certs\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.923498 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.923474    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-dshm\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:56.987400 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:56.987365    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7s27\" (UniqueName: \"kubernetes.io/projected/ce7b021f-cc78-4f81-9160-dedfde5efa2e-kube-api-access-q7s27\") pod \"distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") " pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:57.078580 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:57.078479    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"]
Nov 02 15:16:57.084836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:57.084809    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43407818-rrjkn"]
Nov 02 15:16:57.090163 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:57.090142    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:57.573354 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:16:57.572609    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce7b021f_cc78_4f81_9160_dedfde5efa2e.slice/crio-d51c820b91660736882fe02b8a9caec6810070d61b52591f18b9af9ed212a9e5 WatchSource:0}: Error finding container d51c820b91660736882fe02b8a9caec6810070d61b52591f18b9af9ed212a9e5: Status 404 returned error can't find the container with id d51c820b91660736882fe02b8a9caec6810070d61b52591f18b9af9ed212a9e5
Nov 02 15:16:57.573649 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:57.573628    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"]
Nov 02 15:16:57.871941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:57.871856    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 15:16:57.871941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:57.871917    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 15:16:57.927933 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:57.927899    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" event={"ID":"ce7b021f-cc78-4f81-9160-dedfde5efa2e","Type":"ContainerStarted","Data":"b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57"}
Nov 02 15:16:57.928065 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:57.927942    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" event={"ID":"ce7b021f-cc78-4f81-9160-dedfde5efa2e","Type":"ContainerStarted","Data":"d51c820b91660736882fe02b8a9caec6810070d61b52591f18b9af9ed212a9e5"}
Nov 02 15:16:58.071890 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:58.071831    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podStartSLOduration=2.071814324 podStartE2EDuration="2.071814324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:58.044386206 +0000 UTC m=+1225.637003007" watchObservedRunningTime="2026-11-02 15:16:58.071814324 +0000 UTC m=+1225.664431123"
Nov 02 15:16:58.090812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:58.090780    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:16:58.522729 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:58.522695    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975e03dd-1501-4a43-ac9b-0c9beba3cf0d" path="/var/lib/kubelet/pods/975e03dd-1501-4a43-ac9b-0c9beba3cf0d/volumes"
Nov 02 15:16:58.872156 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:58.872076    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 15:16:58.872156 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:58.872131    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 15:16:59.908047 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:59.908012    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:59.908897 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:59.908862    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:16:59.910132 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:16:59.910100    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:17:02.070151 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:02.070108    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"]
Nov 02 15:17:02.120851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:02.120819    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc04803-b6k6z"]
Nov 02 15:17:02.539410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:02.539369    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6413a9f1-1ab5-4970-a969-f343db4dbd7d" path="/var/lib/kubelet/pods/6413a9f1-1ab5-4970-a969-f343db4dbd7d/volumes"
Nov 02 15:17:03.059957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.059921    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"]
Nov 02 15:17:03.110694 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.110652    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d14500a-gpggr"]
Nov 02 15:17:03.524609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.524571    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"]
Nov 02 15:17:03.530402 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.530374    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:03.583489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.583432    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:17:03.633269 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.633228    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:17:03.633744 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.633720    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:17:03.633849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.633774    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:17:03.638438 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.638413    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"]
Nov 02 15:17:03.650632 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.650603    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8994365a-k4z28\" (UID: \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:03.650785 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.650768    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8994365a-k4z28\" (UID: \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:03.650847 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.650838    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6h8m\" (UniqueName: \"kubernetes.io/projected/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-kube-api-access-x6h8m\") pod \"maas-api-revoke-sub-8994365a-k4z28\" (UID: \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:03.751869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.751826    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8994365a-k4z28\" (UID: \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:03.752052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.751948    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8994365a-k4z28\" (UID: \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:03.752052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.752004    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6h8m\" (UniqueName: \"kubernetes.io/projected/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-kube-api-access-x6h8m\") pod \"maas-api-revoke-sub-8994365a-k4z28\" (UID: \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:03.752653 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.752627    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8994365a-k4z28\" (UID: \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:03.754224 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.754202    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8994365a-k4z28\" (UID: \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:03.813704 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.813635    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6h8m\" (UniqueName: \"kubernetes.io/projected/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-kube-api-access-x6h8m\") pod \"maas-api-revoke-sub-8994365a-k4z28\" (UID: \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:03.846180 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:03.846144    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:04.119610 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.119540    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"]
Nov 02 15:17:04.124845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.124825    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:04.186514 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.186476    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"]
Nov 02 15:17:04.236753 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.236720    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"]
Nov 02 15:17:04.237235 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:04.237213    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc9aa5a_b8a5_4d64_9d78_0209730f77f5.slice/crio-49fb7850bb80ddf377e7f8f4312c6fde7cb4c08b268b18c7d938840cecc8f973 WatchSource:0}: Error finding container 49fb7850bb80ddf377e7f8f4312c6fde7cb4c08b268b18c7d938840cecc8f973: Status 404 returned error can't find the container with id 49fb7850bb80ddf377e7f8f4312c6fde7cb4c08b268b18c7d938840cecc8f973
Nov 02 15:17:04.257839 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.257818    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35212288-ebce-42b4-beab-807ca74c1562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d271c8a-xm727\" (UID: \"35212288-ebce-42b4-beab-807ca74c1562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:04.257904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.257885    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35212288-ebce-42b4-beab-807ca74c1562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d271c8a-xm727\" (UID: \"35212288-ebce-42b4-beab-807ca74c1562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:04.258072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.258051    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvvbn\" (UniqueName: \"kubernetes.io/projected/35212288-ebce-42b4-beab-807ca74c1562-kube-api-access-dvvbn\") pod \"maas-api-revoke-sub-7d271c8a-xm727\" (UID: \"35212288-ebce-42b4-beab-807ca74c1562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:04.359649 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.359616    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35212288-ebce-42b4-beab-807ca74c1562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d271c8a-xm727\" (UID: \"35212288-ebce-42b4-beab-807ca74c1562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:04.359816 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.359707    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35212288-ebce-42b4-beab-807ca74c1562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d271c8a-xm727\" (UID: \"35212288-ebce-42b4-beab-807ca74c1562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:04.359901 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.359869    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvvbn\" (UniqueName: \"kubernetes.io/projected/35212288-ebce-42b4-beab-807ca74c1562-kube-api-access-dvvbn\") pod \"maas-api-revoke-sub-7d271c8a-xm727\" (UID: \"35212288-ebce-42b4-beab-807ca74c1562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:04.360295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.360274    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35212288-ebce-42b4-beab-807ca74c1562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d271c8a-xm727\" (UID: \"35212288-ebce-42b4-beab-807ca74c1562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:04.362086 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.362064    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35212288-ebce-42b4-beab-807ca74c1562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d271c8a-xm727\" (UID: \"35212288-ebce-42b4-beab-807ca74c1562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:04.421603 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.421529    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvvbn\" (UniqueName: \"kubernetes.io/projected/35212288-ebce-42b4-beab-807ca74c1562-kube-api-access-dvvbn\") pod \"maas-api-revoke-sub-7d271c8a-xm727\" (UID: \"35212288-ebce-42b4-beab-807ca74c1562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:04.439033 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.439002    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:04.524777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.524745    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30570f1a-79a9-4468-acb0-1dccc3530e5e" path="/var/lib/kubelet/pods/30570f1a-79a9-4468-acb0-1dccc3530e5e/volumes"
Nov 02 15:17:04.827563 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:04.827525    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35212288_ebce_42b4_beab_807ca74c1562.slice/crio-391b6ddafb4914b4f9ac8dc526045f46de8164a0ce401d7f7a35e3947a3e9420 WatchSource:0}: Error finding container 391b6ddafb4914b4f9ac8dc526045f46de8164a0ce401d7f7a35e3947a3e9420: Status 404 returned error can't find the container with id 391b6ddafb4914b4f9ac8dc526045f46de8164a0ce401d7f7a35e3947a3e9420
Nov 02 15:17:04.827735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:04.827630    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"]
Nov 02 15:17:05.029724 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:05.029676    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28" event={"ID":"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5","Type":"ContainerStarted","Data":"1961e527f9b37c5017cc2149ad74ee5bccb91b4997ebe765c9184c22f799ad1c"}
Nov 02 15:17:05.029724 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:05.029730    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28" event={"ID":"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5","Type":"ContainerStarted","Data":"49fb7850bb80ddf377e7f8f4312c6fde7cb4c08b268b18c7d938840cecc8f973"}
Nov 02 15:17:05.031061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:05.031024    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727" event={"ID":"35212288-ebce-42b4-beab-807ca74c1562","Type":"ContainerStarted","Data":"391b6ddafb4914b4f9ac8dc526045f46de8164a0ce401d7f7a35e3947a3e9420"}
Nov 02 15:17:05.148172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:05.148058    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28" podStartSLOduration=2.148041974 podStartE2EDuration="2.148041974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:05.145617045 +0000 UTC m=+1232.738233844" watchObservedRunningTime="2026-11-02 15:17:05.148041974 +0000 UTC m=+1232.740658786"
Nov 02 15:17:06.040120 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.040078    4891 generic.go:356] "Generic (PLEG): container finished" podID="ebc9aa5a-b8a5-4d64-9d78-0209730f77f5" containerID="1961e527f9b37c5017cc2149ad74ee5bccb91b4997ebe765c9184c22f799ad1c" exitCode=0
Nov 02 15:17:06.040349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.040168    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28" event={"ID":"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5","Type":"ContainerDied","Data":"1961e527f9b37c5017cc2149ad74ee5bccb91b4997ebe765c9184c22f799ad1c"}
Nov 02 15:17:06.041866 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.041841    4891 generic.go:356] "Generic (PLEG): container finished" podID="35212288-ebce-42b4-beab-807ca74c1562" containerID="69474b05e2a2f29c86d29c4e855af89c2f3f145cdcc53fe89d16515f70fa98a1" exitCode=0
Nov 02 15:17:06.041990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.041912    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727" event={"ID":"35212288-ebce-42b4-beab-807ca74c1562","Type":"ContainerDied","Data":"69474b05e2a2f29c86d29c4e855af89c2f3f145cdcc53fe89d16515f70fa98a1"}
Nov 02 15:17:06.674509 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.674472    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"]
Nov 02 15:17:06.678627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.678608    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.791325 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.791283    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"]
Nov 02 15:17:06.802197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.802164    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.802378 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.802288    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztrjj\" (UniqueName: \"kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-kube-api-access-ztrjj\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.802448 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.802420    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.802505 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.802465    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.802615 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.802602    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.802656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.802623    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.802656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.802640    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.802757 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.802660    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.802822 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.802803    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.904200 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.904165    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztrjj\" (UniqueName: \"kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-kube-api-access-ztrjj\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.904378 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.904249    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.904378 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.904287    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.904515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.904382    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.904515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.904409    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.904515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.904431    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.904515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.904456    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.904723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.904522    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.904723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.904588    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.904723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.904678    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-certs\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.904878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.904749    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-data\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.905031 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.905008    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.905146 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.905065    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-credential-socket\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.905221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.905198    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istiod-ca-cert\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.906916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.906896    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-envoy\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.907151 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.907128    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-podinfo\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.968692 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.968622    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-token\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.970267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.970235    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztrjj\" (UniqueName: \"kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-kube-api-access-ztrjj\") pod \"e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") " pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:06.992303 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:06.992280    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:07.091071 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.091035    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:17:07.110783 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.110616    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:17:07.252847 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.252820    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:07.257515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.257494    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:07.411713 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.411681    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35212288-ebce-42b4-beab-807ca74c1562-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35212288-ebce-42b4-beab-807ca74c1562-maas-api-openshift-ca\") pod \"35212288-ebce-42b4-beab-807ca74c1562\" (UID: \"35212288-ebce-42b4-beab-807ca74c1562\") "
Nov 02 15:17:07.411888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.411816    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35212288-ebce-42b4-beab-807ca74c1562-kube-api-access-dvvbn\" (UniqueName: \"kubernetes.io/projected/35212288-ebce-42b4-beab-807ca74c1562-kube-api-access-dvvbn\") pod \"35212288-ebce-42b4-beab-807ca74c1562\" (UID: \"35212288-ebce-42b4-beab-807ca74c1562\") "
Nov 02 15:17:07.411950 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.411914    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-kube-api-access-x6h8m\" (UniqueName: \"kubernetes.io/projected/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-kube-api-access-x6h8m\") pod \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\" (UID: \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\") "
Nov 02 15:17:07.412003 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.411956    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-openshift-ca\") pod \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\" (UID: \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\") "
Nov 02 15:17:07.412003 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.411988    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-xks-ca\") pod \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\" (UID: \"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5\") "
Nov 02 15:17:07.412105 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.412029    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35212288-ebce-42b4-beab-807ca74c1562-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35212288-ebce-42b4-beab-807ca74c1562-maas-api-xks-ca\") pod \"35212288-ebce-42b4-beab-807ca74c1562\" (UID: \"35212288-ebce-42b4-beab-807ca74c1562\") "
Nov 02 15:17:07.412162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.412104    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35212288-ebce-42b4-beab-807ca74c1562-maas-api-openshift-ca" pod "35212288-ebce-42b4-beab-807ca74c1562" (UID: "35212288-ebce-42b4-beab-807ca74c1562"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:07.412389 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.412358    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-openshift-ca" pod "ebc9aa5a-b8a5-4d64-9d78-0209730f77f5" (UID: "ebc9aa5a-b8a5-4d64-9d78-0209730f77f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:07.412628 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.412605    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:07.412720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.412634    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35212288-ebce-42b4-beab-807ca74c1562-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:07.414179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.414155    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35212288-ebce-42b4-beab-807ca74c1562-maas-api-xks-ca" pod "35212288-ebce-42b4-beab-807ca74c1562" (UID: "35212288-ebce-42b4-beab-807ca74c1562"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:07.414496 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.414477    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-xks-ca" pod "ebc9aa5a-b8a5-4d64-9d78-0209730f77f5" (UID: "ebc9aa5a-b8a5-4d64-9d78-0209730f77f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:07.414606 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.414564    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35212288-ebce-42b4-beab-807ca74c1562-kube-api-access-dvvbn" pod "35212288-ebce-42b4-beab-807ca74c1562" (UID: "35212288-ebce-42b4-beab-807ca74c1562"). InnerVolumeSpecName "kube-api-access-dvvbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:07.414795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.414761    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-kube-api-access-x6h8m" pod "ebc9aa5a-b8a5-4d64-9d78-0209730f77f5" (UID: "ebc9aa5a-b8a5-4d64-9d78-0209730f77f5"). InnerVolumeSpecName "kube-api-access-x6h8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:07.449253 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.449227    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"]
Nov 02 15:17:07.449772 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:07.449739    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1cdffd6_0920_40c6_b265_dcd53bdbc95f.slice/crio-66cca988d162276414aff946f72c0db47b9a8d47f0e0ba250064e501b0c5840b WatchSource:0}: Error finding container 66cca988d162276414aff946f72c0db47b9a8d47f0e0ba250064e501b0c5840b: Status 404 returned error can't find the container with id 66cca988d162276414aff946f72c0db47b9a8d47f0e0ba250064e501b0c5840b
Nov 02 15:17:07.452323 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.452279    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:07.452410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.452392    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:07.452458 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.452429    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:07.514722 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.514693    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvvbn\" (UniqueName: \"kubernetes.io/projected/35212288-ebce-42b4-beab-807ca74c1562-kube-api-access-dvvbn\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:07.514863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.514729    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6h8m\" (UniqueName: \"kubernetes.io/projected/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-kube-api-access-x6h8m\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:07.514863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.514747    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:07.514863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:07.514763    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35212288-ebce-42b4-beab-807ca74c1562-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:08.056036 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.055995    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"]
Nov 02 15:17:08.063395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.063367    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dd244b3-bsz9q"]
Nov 02 15:17:08.069524 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.069499    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"
Nov 02 15:17:08.069642 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.069516    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727" event={"ID":"35212288-ebce-42b4-beab-807ca74c1562","Type":"ContainerDied","Data":"391b6ddafb4914b4f9ac8dc526045f46de8164a0ce401d7f7a35e3947a3e9420"}
Nov 02 15:17:08.069642 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.069586    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="391b6ddafb4914b4f9ac8dc526045f46de8164a0ce401d7f7a35e3947a3e9420"
Nov 02 15:17:08.071481 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.071459    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" event={"ID":"f1cdffd6-0920-40c6-b265-dcd53bdbc95f","Type":"ContainerStarted","Data":"7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c"}
Nov 02 15:17:08.071606 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.071496    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" event={"ID":"f1cdffd6-0920-40c6-b265-dcd53bdbc95f","Type":"ContainerStarted","Data":"66cca988d162276414aff946f72c0db47b9a8d47f0e0ba250064e501b0c5840b"}
Nov 02 15:17:08.073147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.073122    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28" event={"ID":"ebc9aa5a-b8a5-4d64-9d78-0209730f77f5","Type":"ContainerDied","Data":"49fb7850bb80ddf377e7f8f4312c6fde7cb4c08b268b18c7d938840cecc8f973"}
Nov 02 15:17:08.073239 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.073168    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49fb7850bb80ddf377e7f8f4312c6fde7cb4c08b268b18c7d938840cecc8f973"
Nov 02 15:17:08.073239 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.073199    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"
Nov 02 15:17:08.090437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.090413    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:17:08.204610 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.204512    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" podStartSLOduration=2.204499602 podStartE2EDuration="2.204499602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:08.197871797 +0000 UTC m=+1235.790488597" watchObservedRunningTime="2026-11-02 15:17:08.204499602 +0000 UTC m=+1235.797116412"
Nov 02 15:17:08.522595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.522556    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e41390-f485-4aa2-8fa6-44e029042712" path="/var/lib/kubelet/pods/a3e41390-f485-4aa2-8fa6-44e029042712/volumes"
Nov 02 15:17:08.993348 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.993246    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:08.995046 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.995018    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:17:08.995174 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:08.995065    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 15:17:09.993098 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:09.993062    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:17:09.993470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:09.993112    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 15:17:10.993411 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:10.993379    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:17:10.993810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:10.993439    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 15:17:11.993254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:11.993220    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:17:11.993427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:11.993296    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 15:17:12.078400 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:12.078363    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"]
Nov 02 15:17:12.116807 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:12.116774    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e0eb47-djzrz"]
Nov 02 15:17:12.522294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:12.522259    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9a7b60-6e32-4f02-a984-f5e472cb7dea" path="/var/lib/kubelet/pods/1a9a7b60-6e32-4f02-a984-f5e472cb7dea/volumes"
Nov 02 15:17:12.992833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:12.992751    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:17:12.992833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:12.992800    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 15:17:13.993007 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:13.992966    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:17:13.993559 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:13.993018    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 15:17:14.274578 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:17:14.274540    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:17:14.997002 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:14.996971    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:14.997493 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:14.997344    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:14.997950 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:14.997934    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:17:16.449302 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.449259    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"]
Nov 02 15:17:16.450039 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.450015    4891 cpu_manager.go:414] "containerMap: removing container" podUID="35212288-ebce-42b4-beab-807ca74c1562" containerName="revoke-keys"
Nov 02 15:17:16.450142 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.450041    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35212288-ebce-42b4-beab-807ca74c1562" containerName="revoke-keys"
Nov 02 15:17:16.450142 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.450084    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ebc9aa5a-b8a5-4d64-9d78-0209730f77f5" containerName="revoke-keys"
Nov 02 15:17:16.450142 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.450093    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc9aa5a-b8a5-4d64-9d78-0209730f77f5" containerName="revoke-keys"
Nov 02 15:17:16.450285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.450263    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="35212288-ebce-42b4-beab-807ca74c1562" containerName="revoke-keys"
Nov 02 15:17:16.450371 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.450284    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc9aa5a-b8a5-4d64-9d78-0209730f77f5" containerName="revoke-keys"
Nov 02 15:17:16.455064 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.455041    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:16.513089 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.513054    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"]
Nov 02 15:17:16.555266 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.555227    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:17:16.555591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.555569    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfn5n\" (UniqueName: \"kubernetes.io/projected/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-kube-api-access-gfn5n\") pod \"maas-api-revoke-sub-ca399e8f-xnp7f\" (UID: \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:16.555655 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.555623    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca399e8f-xnp7f\" (UID: \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:16.555812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.555793    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca399e8f-xnp7f\" (UID: \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:16.558599 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.558578    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:17:16.558599 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.558579    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:17:16.558751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.558620    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:17:16.657339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.657291    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfn5n\" (UniqueName: \"kubernetes.io/projected/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-kube-api-access-gfn5n\") pod \"maas-api-revoke-sub-ca399e8f-xnp7f\" (UID: \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:16.657510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.657375    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca399e8f-xnp7f\" (UID: \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:16.657510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.657478    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca399e8f-xnp7f\" (UID: \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:16.658079 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.658056    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca399e8f-xnp7f\" (UID: \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:16.659924 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.659901    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca399e8f-xnp7f\" (UID: \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:16.718496 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.718434    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfn5n\" (UniqueName: \"kubernetes.io/projected/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-kube-api-access-gfn5n\") pod \"maas-api-revoke-sub-ca399e8f-xnp7f\" (UID: \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:16.769471 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:16.769441    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:17.151681 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:17.151629    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9a34f5_9b4e_4ea1_b3f6_ef123fc2275b.slice/crio-1b971154cdd530dbc4ab988b9d3e1f12fd94553cf8366b21931b84cb6f472d26 WatchSource:0}: Error finding container 1b971154cdd530dbc4ab988b9d3e1f12fd94553cf8366b21931b84cb6f472d26: Status 404 returned error can't find the container with id 1b971154cdd530dbc4ab988b9d3e1f12fd94553cf8366b21931b84cb6f472d26
Nov 02 15:17:17.152999 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:17.152971    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"]
Nov 02 15:17:18.054158 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:18.054124    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"]
Nov 02 15:17:18.104524 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:18.104489    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99d039c-b6t7b"]
Nov 02 15:17:18.109559 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:18.109531    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"]
Nov 02 15:17:18.114045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:18.114023    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096d7a10-gbfmd"]
Nov 02 15:17:18.164160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:18.164130    4891 generic.go:356] "Generic (PLEG): container finished" podID="5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b" containerID="98620c90e704c462736bcb5ce16b7f90671fdd31bd7a23761377ffcf281f81ce" exitCode=0
Nov 02 15:17:18.164443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:18.164222    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f" event={"ID":"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b","Type":"ContainerDied","Data":"98620c90e704c462736bcb5ce16b7f90671fdd31bd7a23761377ffcf281f81ce"}
Nov 02 15:17:18.164443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:18.164283    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f" event={"ID":"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b","Type":"ContainerStarted","Data":"1b971154cdd530dbc4ab988b9d3e1f12fd94553cf8366b21931b84cb6f472d26"}
Nov 02 15:17:18.523689 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:18.523639    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d323de6-da2b-4a24-a19f-7faa80858c6e" path="/var/lib/kubelet/pods/0d323de6-da2b-4a24-a19f-7faa80858c6e/volumes"
Nov 02 15:17:18.524263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:18.524238    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aeef115-33ea-4680-8e02-089853d0709f" path="/var/lib/kubelet/pods/1aeef115-33ea-4680-8e02-089853d0709f/volumes"
Nov 02 15:17:19.051748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.051716    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"]
Nov 02 15:17:19.101119 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.101089    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec3d0f8-tkc9m"]
Nov 02 15:17:19.349028 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.349002    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:19.396399 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.396363    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-kube-api-access-gfn5n\" (UniqueName: \"kubernetes.io/projected/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-kube-api-access-gfn5n\") pod \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\" (UID: \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\") "
Nov 02 15:17:19.396570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.396554    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-xks-ca\") pod \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\" (UID: \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\") "
Nov 02 15:17:19.396624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.396582    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-openshift-ca\") pod \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\" (UID: \"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b\") "
Nov 02 15:17:19.397041 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.397005    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-openshift-ca" pod "5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b" (UID: "5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:19.398500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.398478    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-kube-api-access-gfn5n" pod "5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b" (UID: "5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b"). InnerVolumeSpecName "kube-api-access-gfn5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:19.398609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.398552    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-xks-ca" pod "5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b" (UID: "5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:19.498369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.498329    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:19.498369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.498372    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:19.498595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.498388    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfn5n\" (UniqueName: \"kubernetes.io/projected/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b-kube-api-access-gfn5n\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:19.666961 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.666874    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"]
Nov 02 15:17:19.667667 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.667634    4891 cpu_manager.go:414] "containerMap: removing container" podUID="5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b" containerName="revoke-keys"
Nov 02 15:17:19.667667 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.667666    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b" containerName="revoke-keys"
Nov 02 15:17:19.667851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.667820    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b" containerName="revoke-keys"
Nov 02 15:17:19.671370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.671305    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:19.686008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.685982    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"]
Nov 02 15:17:19.690684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.690664    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:19.700245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.700214    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17ae12f8-bn8c8\" (UID: \"f278d76b-0fae-4736-8073-9396d0228f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:19.700449 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.700428    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ppc9\" (UniqueName: \"kubernetes.io/projected/f278d76b-0fae-4736-8073-9396d0228f10-kube-api-access-5ppc9\") pod \"maas-api-revoke-sub-17ae12f8-bn8c8\" (UID: \"f278d76b-0fae-4736-8073-9396d0228f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:19.700547 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.700521    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17ae12f8-bn8c8\" (UID: \"f278d76b-0fae-4736-8073-9396d0228f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:19.729028 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.728998    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"]
Nov 02 15:17:19.770345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.770284    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"]
Nov 02 15:17:19.801517 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.801480    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0621a2db-m2w6p\" (UID: \"892a2e8d-6377-4b85-bc3e-198b97e39198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:19.801753 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.801546    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17ae12f8-bn8c8\" (UID: \"f278d76b-0fae-4736-8073-9396d0228f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:19.801753 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.801620    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n57k\" (UniqueName: \"kubernetes.io/projected/892a2e8d-6377-4b85-bc3e-198b97e39198-kube-api-access-5n57k\") pod \"maas-api-revoke-sub-0621a2db-m2w6p\" (UID: \"892a2e8d-6377-4b85-bc3e-198b97e39198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:19.801900 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.801774    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ppc9\" (UniqueName: \"kubernetes.io/projected/f278d76b-0fae-4736-8073-9396d0228f10-kube-api-access-5ppc9\") pod \"maas-api-revoke-sub-17ae12f8-bn8c8\" (UID: \"f278d76b-0fae-4736-8073-9396d0228f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:19.801900 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.801831    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0621a2db-m2w6p\" (UID: \"892a2e8d-6377-4b85-bc3e-198b97e39198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:19.801900 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.801889    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17ae12f8-bn8c8\" (UID: \"f278d76b-0fae-4736-8073-9396d0228f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:19.802929 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.802896    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17ae12f8-bn8c8\" (UID: \"f278d76b-0fae-4736-8073-9396d0228f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:19.805733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.805704    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17ae12f8-bn8c8\" (UID: \"f278d76b-0fae-4736-8073-9396d0228f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:19.863812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.863784    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ppc9\" (UniqueName: \"kubernetes.io/projected/f278d76b-0fae-4736-8073-9396d0228f10-kube-api-access-5ppc9\") pod \"maas-api-revoke-sub-17ae12f8-bn8c8\" (UID: \"f278d76b-0fae-4736-8073-9396d0228f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:19.903695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.903663    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0621a2db-m2w6p\" (UID: \"892a2e8d-6377-4b85-bc3e-198b97e39198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:19.903869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.903795    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0621a2db-m2w6p\" (UID: \"892a2e8d-6377-4b85-bc3e-198b97e39198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:19.903869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.903824    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n57k\" (UniqueName: \"kubernetes.io/projected/892a2e8d-6377-4b85-bc3e-198b97e39198-kube-api-access-5n57k\") pod \"maas-api-revoke-sub-0621a2db-m2w6p\" (UID: \"892a2e8d-6377-4b85-bc3e-198b97e39198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:19.904530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.904507    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0621a2db-m2w6p\" (UID: \"892a2e8d-6377-4b85-bc3e-198b97e39198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:19.906216 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.906196    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0621a2db-m2w6p\" (UID: \"892a2e8d-6377-4b85-bc3e-198b97e39198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:19.975809 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.975736    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n57k\" (UniqueName: \"kubernetes.io/projected/892a2e8d-6377-4b85-bc3e-198b97e39198-kube-api-access-5n57k\") pod \"maas-api-revoke-sub-0621a2db-m2w6p\" (UID: \"892a2e8d-6377-4b85-bc3e-198b97e39198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:19.989845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:19.989818    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:20.005011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.004987    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:20.103559 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.103526    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"]
Nov 02 15:17:20.112288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.111429    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.153183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.153151    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"]
Nov 02 15:17:20.158415 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.158392    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.180927 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.180892    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"]
Nov 02 15:17:20.186423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.186392    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"
Nov 02 15:17:20.188472 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.188442    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f" event={"ID":"5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b","Type":"ContainerDied","Data":"1b971154cdd530dbc4ab988b9d3e1f12fd94553cf8366b21931b84cb6f472d26"}
Nov 02 15:17:20.188630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.188495    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b971154cdd530dbc4ab988b9d3e1f12fd94553cf8366b21931b84cb6f472d26"
Nov 02 15:17:20.208653 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.208626    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f752r\" (UniqueName: \"kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-kube-api-access-f752r\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.208775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.208677    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.208775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.208743    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.208888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.208774    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.209109 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209090    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.209221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209126    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.209288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209216    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.209288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209273    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.209416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209303    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.209416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209374    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt24c\" (UniqueName: \"kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-kube-api-access-wt24c\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.209527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209430    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.209527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209468    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.209656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209524    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.209656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209588    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.209656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209614    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.209833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209670    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625e325c-1d99-4f2d-992a-e9abd72792e3-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.209833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209701    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.209833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.209811    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.229370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.225081    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"]
Nov 02 15:17:20.310801 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.310766    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.310991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.310811    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.310991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.310859    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.310991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.310887    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.310991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.310907    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.311223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311027    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt24c\" (UniqueName: \"kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-kube-api-access-wt24c\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.311223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311083    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.311223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311127    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.311223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311165    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.311223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311208    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.311507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311234    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.311507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311256    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.311507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311271    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625e325c-1d99-4f2d-992a-e9abd72792e3-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.311507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311256    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.311507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311286    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.311507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311305    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.311766 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311671    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-credential-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.311982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311956    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.312452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.311995    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625e325c-1d99-4f2d-992a-e9abd72792e3-istiod-ca-cert\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.312571 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.312016    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.312571 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.312293    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-socket\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.312571 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.312352    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-data\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.312571 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.312563    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.312776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.312594    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f752r\" (UniqueName: \"kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-kube-api-access-f752r\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.312776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.312670    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.312776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.312747    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.312939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.312782    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.312939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.312848    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-certs\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.314209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.314183    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.314491 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.314470    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.314930 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.314908    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-envoy\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.315336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.315290    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-podinfo\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.372596 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.372562    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.374043 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.374011    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-token\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.375867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.375839    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f752r\" (UniqueName: \"kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-kube-api-access-f752r\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.382085 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.380286    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt24c\" (UniqueName: \"kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-kube-api-access-wt24c\") pod \"e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") " pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.383563 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.383537    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"]
Nov 02 15:17:20.384147 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:20.384119    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf278d76b_0fae_4736_8073_9396d0228f10.slice/crio-f737536030640ef4f2cdf6b7a5699dada2827ef3b66789e3290914fab85dbfcf WatchSource:0}: Error finding container f737536030640ef4f2cdf6b7a5699dada2827ef3b66789e3290914fab85dbfcf: Status 404 returned error can't find the container with id f737536030640ef4f2cdf6b7a5699dada2827ef3b66789e3290914fab85dbfcf
Nov 02 15:17:20.432041 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.432016    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:20.433661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.433633    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"]
Nov 02 15:17:20.433882 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:20.433840    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892a2e8d_6377_4b85_bc3e_198b97e39198.slice/crio-64cf220afbcc3a75c3ecb9062b53a94046c59fba47e132b71b285f90b7ca1cec WatchSource:0}: Error finding container 64cf220afbcc3a75c3ecb9062b53a94046c59fba47e132b71b285f90b7ca1cec: Status 404 returned error can't find the container with id 64cf220afbcc3a75c3ecb9062b53a94046c59fba47e132b71b285f90b7ca1cec
Nov 02 15:17:20.472726 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.472689    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:20.525181 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.525152    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6" path="/var/lib/kubelet/pods/0d844a2a-cfa1-4a8c-8b42-4c33acbcceb6/volumes"
Nov 02 15:17:20.827853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.827777    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"]
Nov 02 15:17:20.831960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.831936    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:20.898771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.898724    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"]
Nov 02 15:17:20.922013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.921977    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:20.922185 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.922159    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:20.922272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.922216    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee0893d-8caa-4d67-971c-d51bd8955783-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:20.922353 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.922265    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:20.922418 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.922363    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:20.922473 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.922430    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xnh6\" (UniqueName: \"kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-kube-api-access-7xnh6\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:20.922584 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.922558    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:20.924679 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.922607    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:20.924679 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.922667    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:20.935517 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:20.935480    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7aa2fe0_c2d3_467e_938c_0379eedc35a5.slice/crio-eb31f9d70c95923e6d60a8949e83b7c2790547c82dd6d5365d6c400a8c4d200f WatchSource:0}: Error finding container eb31f9d70c95923e6d60a8949e83b7c2790547c82dd6d5365d6c400a8c4d200f: Status 404 returned error can't find the container with id eb31f9d70c95923e6d60a8949e83b7c2790547c82dd6d5365d6c400a8c4d200f
Nov 02 15:17:20.935517 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.935504    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"]
Nov 02 15:17:20.938430 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.938394    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:20.938507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.938476    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:20.938558 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.938522    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:20.973346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.973306    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"]
Nov 02 15:17:20.985629 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:20.985601    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625e325c_1d99_4f2d_992a_e9abd72792e3.slice/crio-21733ab8a4b8f58f80938a95a1b2b3d10c2f338f6d2ecc96b921765332d6ab82 WatchSource:0}: Error finding container 21733ab8a4b8f58f80938a95a1b2b3d10c2f338f6d2ecc96b921765332d6ab82: Status 404 returned error can't find the container with id 21733ab8a4b8f58f80938a95a1b2b3d10c2f338f6d2ecc96b921765332d6ab82
Nov 02 15:17:20.988614 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.988580    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:20.988708 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.988655    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:20.988708 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:20.988688    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:21.023849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.023816    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.023977 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.023963    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.024030 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.023996    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee0893d-8caa-4d67-971c-d51bd8955783-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.024083 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.024039    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.024083 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.024068    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.024362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.024126    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xnh6\" (UniqueName: \"kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-kube-api-access-7xnh6\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.024362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.024197    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.024362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.024232    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.024362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.024292    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.024581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.024404    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-certs\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.024876 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.024843    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-credential-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.024960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.024880    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-socket\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.024960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.024861    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee0893d-8caa-4d67-971c-d51bd8955783-istiod-ca-cert\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.024960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.024939    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-data\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.027190 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.027152    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-podinfo\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.027273 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.027256    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-envoy\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.089851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.089736    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-token\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.092346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.092098    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xnh6\" (UniqueName: \"kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-kube-api-access-7xnh6\") pod \"e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") " pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.145862 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.145610    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:21.204345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.201036    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" event={"ID":"625e325c-1d99-4f2d-992a-e9abd72792e3","Type":"ContainerStarted","Data":"4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f"}
Nov 02 15:17:21.204345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.201101    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" event={"ID":"625e325c-1d99-4f2d-992a-e9abd72792e3","Type":"ContainerStarted","Data":"21733ab8a4b8f58f80938a95a1b2b3d10c2f338f6d2ecc96b921765332d6ab82"}
Nov 02 15:17:21.207297 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.206831    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p" event={"ID":"892a2e8d-6377-4b85-bc3e-198b97e39198","Type":"ContainerStarted","Data":"1723035ec8d5d46c091856c957dd481f7aa3b956be1e55f3897c8a91e6c2cfb8"}
Nov 02 15:17:21.207297 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.206880    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p" event={"ID":"892a2e8d-6377-4b85-bc3e-198b97e39198","Type":"ContainerStarted","Data":"64cf220afbcc3a75c3ecb9062b53a94046c59fba47e132b71b285f90b7ca1cec"}
Nov 02 15:17:21.208774 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.208748    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" event={"ID":"d7aa2fe0-c2d3-467e-938c-0379eedc35a5","Type":"ContainerStarted","Data":"cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d"}
Nov 02 15:17:21.208912 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.208792    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" event={"ID":"d7aa2fe0-c2d3-467e-938c-0379eedc35a5","Type":"ContainerStarted","Data":"eb31f9d70c95923e6d60a8949e83b7c2790547c82dd6d5365d6c400a8c4d200f"}
Nov 02 15:17:21.212021 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.211995    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8" event={"ID":"f278d76b-0fae-4736-8073-9396d0228f10","Type":"ContainerStarted","Data":"8bbb993535230a5039fbfe08f80d2f66c210a9b4da57bb5694ba09f3c94f7e33"}
Nov 02 15:17:21.212121 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.212045    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8" event={"ID":"f278d76b-0fae-4736-8073-9396d0228f10","Type":"ContainerStarted","Data":"f737536030640ef4f2cdf6b7a5699dada2827ef3b66789e3290914fab85dbfcf"}
Nov 02 15:17:21.326332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.325598    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" podStartSLOduration=1.325580108 podStartE2EDuration="1.325580108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:21.324220064 +0000 UTC m=+1248.916836863" watchObservedRunningTime="2026-11-02 15:17:21.325580108 +0000 UTC m=+1248.918196911"
Nov 02 15:17:21.432432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.432399    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:21.442358 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.442258    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8" podStartSLOduration=2.442237905 podStartE2EDuration="2.442237905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:21.441451283 +0000 UTC m=+1249.034068083" watchObservedRunningTime="2026-11-02 15:17:21.442237905 +0000 UTC m=+1249.034854705"
Nov 02 15:17:21.473858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.473826    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:21.552155 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.552110    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"]
Nov 02 15:17:21.572198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.571417    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" podStartSLOduration=1.571395705 podStartE2EDuration="1.571395705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:21.565638664 +0000 UTC m=+1249.158255463" watchObservedRunningTime="2026-11-02 15:17:21.571395705 +0000 UTC m=+1249.164012505"
Nov 02 15:17:21.598038 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.597918    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:21.598038 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.598014    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:21.598219 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.598053    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:21.922410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.922268    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p" podStartSLOduration=2.922249637 podStartE2EDuration="2.922249637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:21.691807182 +0000 UTC m=+1249.284423983" watchObservedRunningTime="2026-11-02 15:17:21.922249637 +0000 UTC m=+1249.514866459"
Nov 02 15:17:21.923415 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.923391    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"]
Nov 02 15:17:21.928818 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:21.928795    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:22.017102 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.017063    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"]
Nov 02 15:17:22.137629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.137590    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxlkn\" (UniqueName: \"kubernetes.io/projected/02eeda9c-0ffc-495d-af44-76928db6e20d-kube-api-access-mxlkn\") pod \"maas-api-revoke-sub-b490704d-sv9xw\" (UID: \"02eeda9c-0ffc-495d-af44-76928db6e20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:22.143347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.139947    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b490704d-sv9xw\" (UID: \"02eeda9c-0ffc-495d-af44-76928db6e20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:22.143347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.140012    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b490704d-sv9xw\" (UID: \"02eeda9c-0ffc-495d-af44-76928db6e20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:22.213600 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.213502    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"]
Nov 02 15:17:22.242914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.242856    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b490704d-sv9xw\" (UID: \"02eeda9c-0ffc-495d-af44-76928db6e20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:22.243097 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.242921    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b490704d-sv9xw\" (UID: \"02eeda9c-0ffc-495d-af44-76928db6e20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:22.243097 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.243013    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlkn\" (UniqueName: \"kubernetes.io/projected/02eeda9c-0ffc-495d-af44-76928db6e20d-kube-api-access-mxlkn\") pod \"maas-api-revoke-sub-b490704d-sv9xw\" (UID: \"02eeda9c-0ffc-495d-af44-76928db6e20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:22.244251 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.244206    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b490704d-sv9xw\" (UID: \"02eeda9c-0ffc-495d-af44-76928db6e20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:22.245291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.245258    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b196c434-l9qk9"]
Nov 02 15:17:22.362420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.354430    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b490704d-sv9xw\" (UID: \"02eeda9c-0ffc-495d-af44-76928db6e20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:22.362420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.359055    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxlkn\" (UniqueName: \"kubernetes.io/projected/02eeda9c-0ffc-495d-af44-76928db6e20d-kube-api-access-mxlkn\") pod \"maas-api-revoke-sub-b490704d-sv9xw\" (UID: \"02eeda9c-0ffc-495d-af44-76928db6e20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:22.434337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.432083    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"]
Nov 02 15:17:22.439605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.439244    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 15:17:22.439605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.439300    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 15:17:22.498385 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.479347    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" event={"ID":"5ee0893d-8caa-4d67-971c-d51bd8955783","Type":"ContainerStarted","Data":"d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256"}
Nov 02 15:17:22.499820 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.499791    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 15:17:22.500132 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.500107    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" podUID="625e325c-1d99-4f2d-992a-e9abd72792e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 15:17:22.500257 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.500062    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.531551 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.531493    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" event={"ID":"5ee0893d-8caa-4d67-971c-d51bd8955783","Type":"ContainerStarted","Data":"7351cfad643e9bf821221a26e07bc36a122828395c073b3ae8cbb34f4359389b"}
Nov 02 15:17:22.610344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.586551    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:22.610344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.590018    4891 generic.go:356] "Generic (PLEG): container finished" podID="f278d76b-0fae-4736-8073-9396d0228f10" containerID="8bbb993535230a5039fbfe08f80d2f66c210a9b4da57bb5694ba09f3c94f7e33" exitCode=0
Nov 02 15:17:22.610344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.590797    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.616500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.590887    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.616500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.615783    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.616500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.615962    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.616500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.615939    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" podStartSLOduration=2.6159203 podStartE2EDuration="2.6159203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:22.593196649 +0000 UTC m=+1250.185813450" watchObservedRunningTime="2026-11-02 15:17:22.6159203 +0000 UTC m=+1250.208537101"
Nov 02 15:17:22.616500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.615991    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.616500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.616053    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.616500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.616109    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpgvm\" (UniqueName: \"kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-kube-api-access-qpgvm\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.616500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.616292    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.616500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.616362    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.619219 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.616838    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab3c0aa3-cf80-4687-a90d-48547900c30c" path="/var/lib/kubelet/pods/ab3c0aa3-cf80-4687-a90d-48547900c30c/volumes"
Nov 02 15:17:22.619219 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.617287    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8" event={"ID":"f278d76b-0fae-4736-8073-9396d0228f10","Type":"ContainerDied","Data":"8bbb993535230a5039fbfe08f80d2f66c210a9b4da57bb5694ba09f3c94f7e33"}
Nov 02 15:17:22.619219 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.617387    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"]
Nov 02 15:17:22.626343 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.626121    4891 generic.go:356] "Generic (PLEG): container finished" podID="892a2e8d-6377-4b85-bc3e-198b97e39198" containerID="1723035ec8d5d46c091856c957dd481f7aa3b956be1e55f3897c8a91e6c2cfb8" exitCode=0
Nov 02 15:17:22.629361 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.627472    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p" event={"ID":"892a2e8d-6377-4b85-bc3e-198b97e39198","Type":"ContainerDied","Data":"1723035ec8d5d46c091856c957dd481f7aa3b956be1e55f3897c8a91e6c2cfb8"}
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.720008    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.720072    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.720136    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.720188    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.724873    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.724938    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.725003    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.725057    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgvm\" (UniqueName: \"kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-kube-api-access-qpgvm\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.725290    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.726209    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.730408    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.731293    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.736720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.733785    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.741895 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.741855    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.778397 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.751627    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.778397 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.762654    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.836718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.836649    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.838796 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.838744    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgvm\" (UniqueName: \"kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-kube-api-access-qpgvm\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:22.926873 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:22.926596    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:23.177337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:23.169405    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:23.224410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:23.213467    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:17:23.224410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:23.213528    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:17:23.436349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:23.435715    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 15:17:23.436349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:23.435777    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 15:17:23.475929 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:23.475886    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 15:17:23.476247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:23.476213    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" podUID="625e325c-1d99-4f2d-992a-e9abd72792e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 15:17:23.635160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:23.634152    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"]
Nov 02 15:17:23.635602 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:23.635576    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02eeda9c_0ffc_495d_af44_76928db6e20d.slice/crio-b3155dbbe53a6b4e14aca5aa109341f20f9d931440083ffc979d01a43d49c87c WatchSource:0}: Error finding container b3155dbbe53a6b4e14aca5aa109341f20f9d931440083ffc979d01a43d49c87c: Status 404 returned error can't find the container with id b3155dbbe53a6b4e14aca5aa109341f20f9d931440083ffc979d01a43d49c87c
Nov 02 15:17:24.026849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.026768    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"]
Nov 02 15:17:24.127252 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:24.104435    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4bd3cc_281d_491e_a0c5_8164b864ecc6.slice/crio-ad7ea1c1e30dd202de8f598010ade20c126546a13ead96038bf5dfaf6f798979 WatchSource:0}: Error finding container ad7ea1c1e30dd202de8f598010ade20c126546a13ead96038bf5dfaf6f798979: Status 404 returned error can't find the container with id ad7ea1c1e30dd202de8f598010ade20c126546a13ead96038bf5dfaf6f798979
Nov 02 15:17:24.131345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.129505    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:24.131345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.129607    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:24.131345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.129649    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:24.216627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.216594    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:17:24.216874 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.216843    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:17:24.339296 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.338944    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:24.433028 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.432997    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 15:17:24.433167 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.433138    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 15:17:24.479822 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.479754    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 15:17:24.480023 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.479830    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" podUID="625e325c-1d99-4f2d-992a-e9abd72792e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 15:17:24.525359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.525193    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f278d76b-0fae-4736-8073-9396d0228f10-kube-api-access-5ppc9\" (UniqueName: \"kubernetes.io/projected/f278d76b-0fae-4736-8073-9396d0228f10-kube-api-access-5ppc9\") pod \"f278d76b-0fae-4736-8073-9396d0228f10\" (UID: \"f278d76b-0fae-4736-8073-9396d0228f10\") "
Nov 02 15:17:24.525359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.525256    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-xks-ca\") pod \"f278d76b-0fae-4736-8073-9396d0228f10\" (UID: \"f278d76b-0fae-4736-8073-9396d0228f10\") "
Nov 02 15:17:24.525359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.525351    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-openshift-ca\") pod \"f278d76b-0fae-4736-8073-9396d0228f10\" (UID: \"f278d76b-0fae-4736-8073-9396d0228f10\") "
Nov 02 15:17:24.570335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.543541    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-openshift-ca" pod "f278d76b-0fae-4736-8073-9396d0228f10" (UID: "f278d76b-0fae-4736-8073-9396d0228f10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:24.588604 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.579633    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-xks-ca" pod "f278d76b-0fae-4736-8073-9396d0228f10" (UID: "f278d76b-0fae-4736-8073-9396d0228f10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:24.588604 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.583839    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f278d76b-0fae-4736-8073-9396d0228f10-kube-api-access-5ppc9" pod "f278d76b-0fae-4736-8073-9396d0228f10" (UID: "f278d76b-0fae-4736-8073-9396d0228f10"). InnerVolumeSpecName "kube-api-access-5ppc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:24.674208 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.637116    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ppc9\" (UniqueName: \"kubernetes.io/projected/f278d76b-0fae-4736-8073-9396d0228f10-kube-api-access-5ppc9\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:24.674208 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.637152    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:24.674208 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.637180    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f278d76b-0fae-4736-8073-9396d0228f10-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:24.895666 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.895574    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:24.896266 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.896234    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw" event={"ID":"02eeda9c-0ffc-495d-af44-76928db6e20d","Type":"ContainerStarted","Data":"17d7b766599897906db4303849c8da1b69a5b579e0c6faaab2008512af66f800"}
Nov 02 15:17:24.904548 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.904519    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw" event={"ID":"02eeda9c-0ffc-495d-af44-76928db6e20d","Type":"ContainerStarted","Data":"b3155dbbe53a6b4e14aca5aa109341f20f9d931440083ffc979d01a43d49c87c"}
Nov 02 15:17:24.904681 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.904667    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" event={"ID":"6f4bd3cc-281d-491e-a0c5-8164b864ecc6","Type":"ContainerStarted","Data":"ad7ea1c1e30dd202de8f598010ade20c126546a13ead96038bf5dfaf6f798979"}
Nov 02 15:17:24.914333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.910459    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"
Nov 02 15:17:24.924565 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.924532    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8" event={"ID":"f278d76b-0fae-4736-8073-9396d0228f10","Type":"ContainerDied","Data":"f737536030640ef4f2cdf6b7a5699dada2827ef3b66789e3290914fab85dbfcf"}
Nov 02 15:17:24.924772 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.924758    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f737536030640ef4f2cdf6b7a5699dada2827ef3b66789e3290914fab85dbfcf"
Nov 02 15:17:24.935338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:24.935296    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:25.068372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.068242    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-openshift-ca\") pod \"892a2e8d-6377-4b85-bc3e-198b97e39198\" (UID: \"892a2e8d-6377-4b85-bc3e-198b97e39198\") "
Nov 02 15:17:25.068372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.068296    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-xks-ca\") pod \"892a2e8d-6377-4b85-bc3e-198b97e39198\" (UID: \"892a2e8d-6377-4b85-bc3e-198b97e39198\") "
Nov 02 15:17:25.068631 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.068563    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892a2e8d-6377-4b85-bc3e-198b97e39198-kube-api-access-5n57k\" (UniqueName: \"kubernetes.io/projected/892a2e8d-6377-4b85-bc3e-198b97e39198-kube-api-access-5n57k\") pod \"892a2e8d-6377-4b85-bc3e-198b97e39198\" (UID: \"892a2e8d-6377-4b85-bc3e-198b97e39198\") "
Nov 02 15:17:25.073336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.070182    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-openshift-ca" pod "892a2e8d-6377-4b85-bc3e-198b97e39198" (UID: "892a2e8d-6377-4b85-bc3e-198b97e39198"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:25.073336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.071728    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" podStartSLOduration=3.071708928 podStartE2EDuration="3.071708928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:25.028941648 +0000 UTC m=+1252.621558449" watchObservedRunningTime="2026-11-02 15:17:25.071708928 +0000 UTC m=+1252.664325731"
Nov 02 15:17:25.118333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.116452    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-xks-ca" pod "892a2e8d-6377-4b85-bc3e-198b97e39198" (UID: "892a2e8d-6377-4b85-bc3e-198b97e39198"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:25.118333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.116600    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892a2e8d-6377-4b85-bc3e-198b97e39198-kube-api-access-5n57k" pod "892a2e8d-6377-4b85-bc3e-198b97e39198" (UID: "892a2e8d-6377-4b85-bc3e-198b97e39198"). InnerVolumeSpecName "kube-api-access-5n57k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:25.153247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.152805    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:17:25.153247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.152871    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:17:25.172756 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.172724    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n57k\" (UniqueName: \"kubernetes.io/projected/892a2e8d-6377-4b85-bc3e-198b97e39198-kube-api-access-5n57k\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:25.173011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.172990    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:25.173179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.173162    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892a2e8d-6377-4b85-bc3e-198b97e39198-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:25.311343 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.294975    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw" podStartSLOduration=4.294955313 podStartE2EDuration="4.294955313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:25.16369387 +0000 UTC m=+1252.756310669" watchObservedRunningTime="2026-11-02 15:17:25.294955313 +0000 UTC m=+1252.887572116"
Nov 02 15:17:25.436012 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.435923    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 15:17:25.436595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.436562    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 15:17:25.474052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.474016    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 15:17:25.474343 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.474293    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" podUID="625e325c-1d99-4f2d-992a-e9abd72792e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 15:17:25.937360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.933989    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 15:17:25.937360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:25.934078    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 15:17:26.017255 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.017216    4891 generic.go:356] "Generic (PLEG): container finished" podID="02eeda9c-0ffc-495d-af44-76928db6e20d" containerID="17d7b766599897906db4303849c8da1b69a5b579e0c6faaab2008512af66f800" exitCode=0
Nov 02 15:17:26.017706 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.017675    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw" event={"ID":"02eeda9c-0ffc-495d-af44-76928db6e20d","Type":"ContainerDied","Data":"17d7b766599897906db4303849c8da1b69a5b579e0c6faaab2008512af66f800"}
Nov 02 15:17:26.028179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.028140    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" event={"ID":"6f4bd3cc-281d-491e-a0c5-8164b864ecc6","Type":"ContainerStarted","Data":"f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50"}
Nov 02 15:17:26.043195 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.043159    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p" event={"ID":"892a2e8d-6377-4b85-bc3e-198b97e39198","Type":"ContainerDied","Data":"64cf220afbcc3a75c3ecb9062b53a94046c59fba47e132b71b285f90b7ca1cec"}
Nov 02 15:17:26.043446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.043429    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64cf220afbcc3a75c3ecb9062b53a94046c59fba47e132b71b285f90b7ca1cec"
Nov 02 15:17:26.043912 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.043895    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"
Nov 02 15:17:26.168863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.157856    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:17:26.168863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.157946    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:17:26.438454 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.433662    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 15:17:26.438454 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.433737    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 15:17:26.476795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.476691    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 15:17:26.477087 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.477057    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" podUID="625e325c-1d99-4f2d-992a-e9abd72792e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 15:17:26.740397 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.689980    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:26.809419 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.808792    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 15:17:26.900338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.884974    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Nov 02 15:17:26.923332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:26.901089    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:27.127401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.062626    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 15:17:27.127401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.089751    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 15:17:27.127401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.083227    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:17:27.127401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.083194    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:27.127401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.088641    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:27.127401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.088677    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:17:27.158711 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.158532    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:17:27.158711 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.158598    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:17:27.257336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.256834    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:27.265473 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.265403    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:27.450512 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.450423    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 15:17:27.463451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.463405    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 15:17:27.498960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.495429    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 15:17:27.550454 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.550409    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" podUID="625e325c-1d99-4f2d-992a-e9abd72792e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 15:17:27.710400 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.698361    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:27.710770 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.710734    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:17:27.742422 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.742377    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:17:27.743993 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.743965    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:17:27.815439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.815397    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:27.816491 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.816465    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:17:27.935489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.935452    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 15:17:27.935760 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:27.935728    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 15:17:28.030412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.030277    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:17:28.098763 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.091398    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Nov 02 15:17:28.124335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.118526    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:28.220335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.219464    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:17:28.220335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.219543    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:17:28.229768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.224557    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:17:28.235101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.235057    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:17:28.239941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.239911    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:17:28.281057 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.280958    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:28.281866 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.281842    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:17:28.344358 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.341335    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:17:28.507528 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.507163    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 15:17:28.512628 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.512581    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" podUID="625e325c-1d99-4f2d-992a-e9abd72792e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 15:17:28.513328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.513286    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 15:17:28.513549 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.513529    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 15:17:28.529333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.526483    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:28.546863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.546781    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:17:28.547108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.547092    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:17:28.547253 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.547240    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:17:28.633289 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.633256    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-openshift-ca\") pod \"02eeda9c-0ffc-495d-af44-76928db6e20d\" (UID: \"02eeda9c-0ffc-495d-af44-76928db6e20d\") "
Nov 02 15:17:28.634040 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.633913    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-openshift-ca" pod "02eeda9c-0ffc-495d-af44-76928db6e20d" (UID: "02eeda9c-0ffc-495d-af44-76928db6e20d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:28.634437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.634415    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-xks-ca\") pod \"02eeda9c-0ffc-495d-af44-76928db6e20d\" (UID: \"02eeda9c-0ffc-495d-af44-76928db6e20d\") "
Nov 02 15:17:28.635114 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.635080    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02eeda9c-0ffc-495d-af44-76928db6e20d-kube-api-access-mxlkn\" (UniqueName: \"kubernetes.io/projected/02eeda9c-0ffc-495d-af44-76928db6e20d-kube-api-access-mxlkn\") pod \"02eeda9c-0ffc-495d-af44-76928db6e20d\" (UID: \"02eeda9c-0ffc-495d-af44-76928db6e20d\") "
Nov 02 15:17:28.637123 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.637097    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:28.637123 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.637098    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-xks-ca" pod "02eeda9c-0ffc-495d-af44-76928db6e20d" (UID: "02eeda9c-0ffc-495d-af44-76928db6e20d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:28.638815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.638785    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eeda9c-0ffc-495d-af44-76928db6e20d-kube-api-access-mxlkn" pod "02eeda9c-0ffc-495d-af44-76928db6e20d" (UID: "02eeda9c-0ffc-495d-af44-76928db6e20d"). InnerVolumeSpecName "kube-api-access-mxlkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:28.745334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.743389    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxlkn\" (UniqueName: \"kubernetes.io/projected/02eeda9c-0ffc-495d-af44-76928db6e20d-kube-api-access-mxlkn\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:28.745334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.743413    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02eeda9c-0ffc-495d-af44-76928db6e20d-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:28.928834 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.928750    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 15:17:28.929066 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:28.929040    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 15:17:29.163336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.149667    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 15:17:29.163336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.149727    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 15:17:29.225513 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.217660    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"]
Nov 02 15:17:29.237380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.234900    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679d8220-2q87c"]
Nov 02 15:17:29.586336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.570486    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"
Nov 02 15:17:29.589068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.584384    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw" event={"ID":"02eeda9c-0ffc-495d-af44-76928db6e20d","Type":"ContainerDied","Data":"b3155dbbe53a6b4e14aca5aa109341f20f9d931440083ffc979d01a43d49c87c"}
Nov 02 15:17:29.589554 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.589530    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3155dbbe53a6b4e14aca5aa109341f20f9d931440083ffc979d01a43d49c87c"
Nov 02 15:17:29.734090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.734036    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:29.735625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.735491    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:29.736549 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.736515    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:17:29.776333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.772629    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:29.776333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.773917    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:29.781647 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.781520    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:17:29.928604 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.928517    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 15:17:29.928604 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:29.928583    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 15:17:30.323360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:30.323302    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:30.325526 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:30.325186    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:30.327229 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:30.327203    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:17:30.522881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:30.522850    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="819d0f4a-890b-4fcf-8059-dfbd30e77644" path="/var/lib/kubelet/pods/819d0f4a-890b-4fcf-8059-dfbd30e77644/volumes"
Nov 02 15:17:30.928855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:30.928819    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 15:17:30.929369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:30.928872    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 15:17:31.928346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:31.928304    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 15:17:31.928515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:31.928369    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 15:17:32.761478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.761448    4891 scope.go:122] "RemoveContainer" containerID="ac457d8919c163c1fcf71d907b92b755ee057010aa70ddda54f8bcd15cc82c8f"
Nov 02 15:17:32.773119 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.773092    4891 scope.go:122] "RemoveContainer" containerID="2324d3f39c2c56bf0c10723a0da92d312194095c1fc4096df207fff613f5aaa6"
Nov 02 15:17:32.783522 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.783477    4891 scope.go:122] "RemoveContainer" containerID="5e217f303bb1f7643c65352e3aa666a529a1cb8282d180fa575ff547a7f87fc8"
Nov 02 15:17:32.795230 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.795207    4891 scope.go:122] "RemoveContainer" containerID="beab5fbea63f3c5abe5a78f9b156079921d1b05e2e0e372ecb9ad0f664373523"
Nov 02 15:17:32.804296 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.804271    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"]
Nov 02 15:17:32.804921 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.804905    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f278d76b-0fae-4736-8073-9396d0228f10" containerName="revoke-keys"
Nov 02 15:17:32.804966 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.804923    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f278d76b-0fae-4736-8073-9396d0228f10" containerName="revoke-keys"
Nov 02 15:17:32.804998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.804970    4891 cpu_manager.go:414] "containerMap: removing container" podUID="892a2e8d-6377-4b85-bc3e-198b97e39198" containerName="revoke-keys"
Nov 02 15:17:32.804998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.804975    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892a2e8d-6377-4b85-bc3e-198b97e39198" containerName="revoke-keys"
Nov 02 15:17:32.804998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.804988    4891 cpu_manager.go:414] "containerMap: removing container" podUID="02eeda9c-0ffc-495d-af44-76928db6e20d" containerName="revoke-keys"
Nov 02 15:17:32.804998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.804993    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02eeda9c-0ffc-495d-af44-76928db6e20d" containerName="revoke-keys"
Nov 02 15:17:32.805123 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.805075    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="892a2e8d-6377-4b85-bc3e-198b97e39198" containerName="revoke-keys"
Nov 02 15:17:32.805123 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.805084    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="02eeda9c-0ffc-495d-af44-76928db6e20d" containerName="revoke-keys"
Nov 02 15:17:32.805123 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.805091    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="f278d76b-0fae-4736-8073-9396d0228f10" containerName="revoke-keys"
Nov 02 15:17:32.810172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.810155    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:32.866946 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.866919    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:17:32.903130 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.903094    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xklbg\" (UniqueName: \"kubernetes.io/projected/4698bf0f-670a-47f6-aa26-670fc0f151a8-kube-api-access-xklbg\") pod \"maas-api-revoke-sub-b59314cf-qhdts\" (UID: \"4698bf0f-670a-47f6-aa26-670fc0f151a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:32.903367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.903350    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b59314cf-qhdts\" (UID: \"4698bf0f-670a-47f6-aa26-670fc0f151a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:32.903462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.903386    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b59314cf-qhdts\" (UID: \"4698bf0f-670a-47f6-aa26-670fc0f151a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:32.916789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.916654    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:17:32.916789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.916716    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:17:32.916789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.916747    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:17:32.921614 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.921589    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"]
Nov 02 15:17:32.928526 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.928503    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 15:17:32.928641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:32.928542    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 15:17:33.008337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:33.008112    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b59314cf-qhdts\" (UID: \"4698bf0f-670a-47f6-aa26-670fc0f151a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:33.008337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:33.008256    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xklbg\" (UniqueName: \"kubernetes.io/projected/4698bf0f-670a-47f6-aa26-670fc0f151a8-kube-api-access-xklbg\") pod \"maas-api-revoke-sub-b59314cf-qhdts\" (UID: \"4698bf0f-670a-47f6-aa26-670fc0f151a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:33.008696 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:33.008564    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b59314cf-qhdts\" (UID: \"4698bf0f-670a-47f6-aa26-670fc0f151a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:33.009492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:33.009462    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b59314cf-qhdts\" (UID: \"4698bf0f-670a-47f6-aa26-670fc0f151a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:33.011260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:33.011239    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b59314cf-qhdts\" (UID: \"4698bf0f-670a-47f6-aa26-670fc0f151a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:33.070466 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:33.070443    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xklbg\" (UniqueName: \"kubernetes.io/projected/4698bf0f-670a-47f6-aa26-670fc0f151a8-kube-api-access-xklbg\") pod \"maas-api-revoke-sub-b59314cf-qhdts\" (UID: \"4698bf0f-670a-47f6-aa26-670fc0f151a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:33.129456 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:33.129418    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:33.528702 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:33.528620    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"]
Nov 02 15:17:33.529583 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:33.529557    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4698bf0f_670a_47f6_aa26_670fc0f151a8.slice/crio-b60891f752f40a1480d864108d702f20f12a51af58b33b6828bafe8c1dd0a9ab WatchSource:0}: Error finding container b60891f752f40a1480d864108d702f20f12a51af58b33b6828bafe8c1dd0a9ab: Status 404 returned error can't find the container with id b60891f752f40a1480d864108d702f20f12a51af58b33b6828bafe8c1dd0a9ab
Nov 02 15:17:33.647265 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:33.647217    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts" event={"ID":"4698bf0f-670a-47f6-aa26-670fc0f151a8","Type":"ContainerStarted","Data":"b60891f752f40a1480d864108d702f20f12a51af58b33b6828bafe8c1dd0a9ab"}
Nov 02 15:17:33.932411 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:33.932381    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:33.932879 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:33.932867    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:33.933676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:33.933657    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:17:34.402390 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.402358    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"]
Nov 02 15:17:34.406550 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.406533    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:34.473784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.473748    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"]
Nov 02 15:17:34.532451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.532415    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40374a74-wrnx8\" (UID: \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:34.532624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.532603    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gks8f\" (UniqueName: \"kubernetes.io/projected/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-kube-api-access-gks8f\") pod \"maas-api-revoke-sub-40374a74-wrnx8\" (UID: \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:34.532819 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.532798    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40374a74-wrnx8\" (UID: \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:34.634651 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.634612    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gks8f\" (UniqueName: \"kubernetes.io/projected/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-kube-api-access-gks8f\") pod \"maas-api-revoke-sub-40374a74-wrnx8\" (UID: \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:34.634836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.634815    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40374a74-wrnx8\" (UID: \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:34.634905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.634887    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40374a74-wrnx8\" (UID: \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:34.635619 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.635594    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40374a74-wrnx8\" (UID: \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:34.637301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.637279    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40374a74-wrnx8\" (UID: \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:34.667097 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.667013    4891 generic.go:356] "Generic (PLEG): container finished" podID="4698bf0f-670a-47f6-aa26-670fc0f151a8" containerID="cc3337d62b6f07117967e5f108c9daa4bc7f2713eb53a96f19e21c3d77f9e436" exitCode=0
Nov 02 15:17:34.667220 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.667090    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts" event={"ID":"4698bf0f-670a-47f6-aa26-670fc0f151a8","Type":"ContainerDied","Data":"cc3337d62b6f07117967e5f108c9daa4bc7f2713eb53a96f19e21c3d77f9e436"}
Nov 02 15:17:34.695779 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.695749    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gks8f\" (UniqueName: \"kubernetes.io/projected/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-kube-api-access-gks8f\") pod \"maas-api-revoke-sub-40374a74-wrnx8\" (UID: \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:34.720631 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:34.720598    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:35.105671 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:35.105639    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef8e0fb_be13_408f_b2f7_69c70e569e0d.slice/crio-93be8a2a301ae3edf3244272bbf7d827dcf2cfdbabefa2cbdef2f97f74ac5620 WatchSource:0}: Error finding container 93be8a2a301ae3edf3244272bbf7d827dcf2cfdbabefa2cbdef2f97f74ac5620: Status 404 returned error can't find the container with id 93be8a2a301ae3edf3244272bbf7d827dcf2cfdbabefa2cbdef2f97f74ac5620
Nov 02 15:17:35.106782 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:35.106752    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"]
Nov 02 15:17:35.679891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:35.679801    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8" event={"ID":"2ef8e0fb-be13-408f-b2f7-69c70e569e0d","Type":"ContainerStarted","Data":"913cada4a089ecb7d8494d2364e9c3936cde039429bd9350064f7f45793b8185"}
Nov 02 15:17:35.679891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:35.679862    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8" event={"ID":"2ef8e0fb-be13-408f-b2f7-69c70e569e0d","Type":"ContainerStarted","Data":"93be8a2a301ae3edf3244272bbf7d827dcf2cfdbabefa2cbdef2f97f74ac5620"}
Nov 02 15:17:35.845332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:35.838813    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8" podStartSLOduration=1.838793254 podStartE2EDuration="1.838793254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:35.835554903 +0000 UTC m=+1263.428171703" watchObservedRunningTime="2026-11-02 15:17:35.838793254 +0000 UTC m=+1263.431410078"
Nov 02 15:17:36.048690 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.045751    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:36.168207 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.166182    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-openshift-ca\") pod \"4698bf0f-670a-47f6-aa26-670fc0f151a8\" (UID: \"4698bf0f-670a-47f6-aa26-670fc0f151a8\") "
Nov 02 15:17:36.168207 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.166324    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4698bf0f-670a-47f6-aa26-670fc0f151a8-kube-api-access-xklbg\" (UniqueName: \"kubernetes.io/projected/4698bf0f-670a-47f6-aa26-670fc0f151a8-kube-api-access-xklbg\") pod \"4698bf0f-670a-47f6-aa26-670fc0f151a8\" (UID: \"4698bf0f-670a-47f6-aa26-670fc0f151a8\") "
Nov 02 15:17:36.168207 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.166544    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-xks-ca\") pod \"4698bf0f-670a-47f6-aa26-670fc0f151a8\" (UID: \"4698bf0f-670a-47f6-aa26-670fc0f151a8\") "
Nov 02 15:17:36.175576 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.174881    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-openshift-ca" pod "4698bf0f-670a-47f6-aa26-670fc0f151a8" (UID: "4698bf0f-670a-47f6-aa26-670fc0f151a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:36.187396 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.183413    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-xks-ca" pod "4698bf0f-670a-47f6-aa26-670fc0f151a8" (UID: "4698bf0f-670a-47f6-aa26-670fc0f151a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:36.187396 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.185611    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4698bf0f-670a-47f6-aa26-670fc0f151a8-kube-api-access-xklbg" pod "4698bf0f-670a-47f6-aa26-670fc0f151a8" (UID: "4698bf0f-670a-47f6-aa26-670fc0f151a8"). InnerVolumeSpecName "kube-api-access-xklbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:36.254829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.251529    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"]
Nov 02 15:17:36.254829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.252470    4891 cpu_manager.go:414] "containerMap: removing container" podUID="4698bf0f-670a-47f6-aa26-670fc0f151a8" containerName="revoke-keys"
Nov 02 15:17:36.254829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.252494    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4698bf0f-670a-47f6-aa26-670fc0f151a8" containerName="revoke-keys"
Nov 02 15:17:36.254829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.252682    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="4698bf0f-670a-47f6-aa26-670fc0f151a8" containerName="revoke-keys"
Nov 02 15:17:36.257093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.257066    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:36.269278 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.269236    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xklbg\" (UniqueName: \"kubernetes.io/projected/4698bf0f-670a-47f6-aa26-670fc0f151a8-kube-api-access-xklbg\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:36.269634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.269612    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:36.269826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.269763    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4698bf0f-670a-47f6-aa26-670fc0f151a8-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:36.298971 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.298928    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"]
Nov 02 15:17:36.371247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.371194    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcs5p\" (UniqueName: \"kubernetes.io/projected/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-kube-api-access-xcs5p\") pod \"maas-api-revoke-sub-75d76dd5-2k68z\" (UID: \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:36.371476 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.371347    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d76dd5-2k68z\" (UID: \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:36.371476 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.371455    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d76dd5-2k68z\" (UID: \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:36.498387 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.493598    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d76dd5-2k68z\" (UID: \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:36.498387 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.493720    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d76dd5-2k68z\" (UID: \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:36.498387 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.493856    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcs5p\" (UniqueName: \"kubernetes.io/projected/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-kube-api-access-xcs5p\") pod \"maas-api-revoke-sub-75d76dd5-2k68z\" (UID: \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:36.498387 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.495637    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d76dd5-2k68z\" (UID: \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:36.501446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.501416    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d76dd5-2k68z\" (UID: \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:36.576354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.575042    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcs5p\" (UniqueName: \"kubernetes.io/projected/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-kube-api-access-xcs5p\") pod \"maas-api-revoke-sub-75d76dd5-2k68z\" (UID: \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:36.603350 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.602536    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:36.731575 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.725062    4891 generic.go:356] "Generic (PLEG): container finished" podID="2ef8e0fb-be13-408f-b2f7-69c70e569e0d" containerID="913cada4a089ecb7d8494d2364e9c3936cde039429bd9350064f7f45793b8185" exitCode=0
Nov 02 15:17:36.735339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.734660    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8" event={"ID":"2ef8e0fb-be13-408f-b2f7-69c70e569e0d","Type":"ContainerDied","Data":"913cada4a089ecb7d8494d2364e9c3936cde039429bd9350064f7f45793b8185"}
Nov 02 15:17:36.738333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.736946    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts" event={"ID":"4698bf0f-670a-47f6-aa26-670fc0f151a8","Type":"ContainerDied","Data":"b60891f752f40a1480d864108d702f20f12a51af58b33b6828bafe8c1dd0a9ab"}
Nov 02 15:17:36.738333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.736997    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b60891f752f40a1480d864108d702f20f12a51af58b33b6828bafe8c1dd0a9ab"
Nov 02 15:17:36.738333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:36.737061    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"
Nov 02 15:17:37.941389 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:37.941253    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.006573 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.006523    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.007383 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.007357    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"]
Nov 02 15:17:38.083643 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.078392    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:17:38.119352 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.114255    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z" event={"ID":"d211982c-f8e5-411d-9e60-ef9f6fe95bb2","Type":"ContainerStarted","Data":"97bba7069728ca68275df77cef039b336a229dc87d119d1bc4e3e0d7a52fce18"}
Nov 02 15:17:38.204383 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.169904    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"]
Nov 02 15:17:38.231013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.217066    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"]
Nov 02 15:17:38.231013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.217195    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.329336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.316166    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t472\" (UniqueName: \"kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-kube-api-access-7t472\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.329336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.316248    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.329336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.316295    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.329336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.316342    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.329336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.316370    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.329336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.316432    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.329336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.316459    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.329336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.316494    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.329336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.316542    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.329336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.317995    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.340384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.335390    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"]
Nov 02 15:17:38.382386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.374557    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28dd6acc-9w25x"]
Nov 02 15:17:38.409283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.409234    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.439700 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.424478    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.440032 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.440004    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.440589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.440569    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.441069 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.441054    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.457503 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.457424    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.458634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.440994    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-data\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.458789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.425489    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istiod-ca-cert\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.458905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.440482    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.459001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.457997    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-certs\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.459113 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.459095    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t472\" (UniqueName: \"kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-kube-api-access-7t472\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.459288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.459269    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.459458 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.459440    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.459583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.459566    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.460166 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.460146    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-credential-socket\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.476394 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.471093    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-podinfo\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.491630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.491574    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-envoy\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.509882 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.509854    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-token\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.550338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.532844    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t472\" (UniqueName: \"kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-kube-api-access-7t472\") pod \"e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") " pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.562336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.559419    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.619336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.618098    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5726fda9-7b7b-4170-a62b-252d7f1d9ebd" path="/var/lib/kubelet/pods/5726fda9-7b7b-4170-a62b-252d7f1d9ebd/volumes"
Nov 02 15:17:38.701809 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.700911    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:38.751473 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.730813    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.973014 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.972975    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.995616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.995563    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:39.000770 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:38.997711    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded"
Nov 02 15:17:39.020769 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.001400    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:17:39.055990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.054439    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:39.113438 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.088806    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:39.210551 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.210495    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:39.229475 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.210922    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:17:39.282393 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.231432    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:39.471504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.471464    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:39.472095 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.472074    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:17:39.521570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.521538    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:39.604642 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.548415    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 15:17:39.604642 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.573409    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:39.604642 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.574353    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:17:39.604642 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.575209    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:39.604642 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.548013    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:17:39.863606 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.863504    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z" event={"ID":"d211982c-f8e5-411d-9e60-ef9f6fe95bb2","Type":"ContainerStarted","Data":"cd83abc56a3f7280b3f6561632ca1dacbfdcacc663a18e783f854889219654af"}
Nov 02 15:17:40.012457 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.012425    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:17:40.015655 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.881355    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-openshift-ca\") pod \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\" (UID: \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\") "
Nov 02 15:17:40.015902 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.015883    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-kube-api-access-gks8f\" (UniqueName: \"kubernetes.io/projected/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-kube-api-access-gks8f\") pod \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\" (UID: \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\") "
Nov 02 15:17:40.016092 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.016074    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-xks-ca\") pod \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\" (UID: \"2ef8e0fb-be13-408f-b2f7-69c70e569e0d\") "
Nov 02 15:17:40.017373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:39.974453    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:40.027357 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.018383    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:40.037189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.027847    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:17:40.062654 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.062616    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:40.063084 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.063066    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:17:40.066215 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:17:40.066188    4891 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd211982c_f8e5_411d_9e60_ef9f6fe95bb2.slice/crio-conmon-cd83abc56a3f7280b3f6561632ca1dacbfdcacc663a18e783f854889219654af.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:17:40.079705 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.079669    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-openshift-ca" pod "2ef8e0fb-be13-408f-b2f7-69c70e569e0d" (UID: "2ef8e0fb-be13-408f-b2f7-69c70e569e0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:40.083265 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.083235    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-kube-api-access-gks8f" pod "2ef8e0fb-be13-408f-b2f7-69c70e569e0d" (UID: "2ef8e0fb-be13-408f-b2f7-69c70e569e0d"). InnerVolumeSpecName "kube-api-access-gks8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:40.092294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.092024    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-xks-ca" pod "2ef8e0fb-be13-408f-b2f7-69c70e569e0d" (UID: "2ef8e0fb-be13-408f-b2f7-69c70e569e0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:40.107751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.107722    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:17:40.125502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.118815    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:40.125502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.118848    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gks8f\" (UniqueName: \"kubernetes.io/projected/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-kube-api-access-gks8f\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:40.125502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.118863    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef8e0fb-be13-408f-b2f7-69c70e569e0d-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:40.125502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.119047    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:17:40.129750 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.129727    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:17:40.181101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.179098    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:17:40.250336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.243448    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:17:40.362963 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.360433    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:40.362963 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.360597    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:40.362963 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.360702    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:40.362963 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.362860    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"]
Nov 02 15:17:40.609775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.609722    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z" podStartSLOduration=4.609705767 podStartE2EDuration="4.609705767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:40.608573939 +0000 UTC m=+1268.201190739" watchObservedRunningTime="2026-11-02 15:17:40.609705767 +0000 UTC m=+1268.202322594"
Nov 02 15:17:40.874586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.874555    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"
Nov 02 15:17:40.874586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.874576    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8" event={"ID":"2ef8e0fb-be13-408f-b2f7-69c70e569e0d","Type":"ContainerDied","Data":"93be8a2a301ae3edf3244272bbf7d827dcf2cfdbabefa2cbdef2f97f74ac5620"}
Nov 02 15:17:40.874832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.874637    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93be8a2a301ae3edf3244272bbf7d827dcf2cfdbabefa2cbdef2f97f74ac5620"
Nov 02 15:17:40.876895 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.876856    4891 generic.go:356] "Generic (PLEG): container finished" podID="d211982c-f8e5-411d-9e60-ef9f6fe95bb2" containerID="cd83abc56a3f7280b3f6561632ca1dacbfdcacc663a18e783f854889219654af" exitCode=0
Nov 02 15:17:40.877019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.876960    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z" event={"ID":"d211982c-f8e5-411d-9e60-ef9f6fe95bb2","Type":"ContainerDied","Data":"cd83abc56a3f7280b3f6561632ca1dacbfdcacc663a18e783f854889219654af"}
Nov 02 15:17:40.878894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.878867    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" event={"ID":"9bee6b2b-9e64-4b60-9f35-0775382a60f1","Type":"ContainerStarted","Data":"8196030bf19326e58de26e793a3a2f39c145d0de5b0fcf900a81a7e048e927b6"}
Nov 02 15:17:40.879034 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:40.878910    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" event={"ID":"9bee6b2b-9e64-4b60-9f35-0775382a60f1","Type":"ContainerStarted","Data":"65fc26e3b088c7471b17aee4b2eeca5747691b38c31ac3a15418e3c534a889c9"}
Nov 02 15:17:41.343665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:41.343605    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" podStartSLOduration=4.343591232 podStartE2EDuration="4.343591232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:41.338748383 +0000 UTC m=+1268.931365184" watchObservedRunningTime="2026-11-02 15:17:41.343591232 +0000 UTC m=+1268.936208031"
Nov 02 15:17:41.732902 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:41.732818    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:41.734244 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:41.734217    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 15:17:41.734374 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:41.734264    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" podUID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 15:17:42.126299 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.123759    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:42.188819 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.188786    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-xks-ca\") pod \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\" (UID: \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\") "
Nov 02 15:17:42.188982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.188835    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-kube-api-access-xcs5p\" (UniqueName: \"kubernetes.io/projected/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-kube-api-access-xcs5p\") pod \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\" (UID: \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\") "
Nov 02 15:17:42.188982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.188875    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-openshift-ca\") pod \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\" (UID: \"d211982c-f8e5-411d-9e60-ef9f6fe95bb2\") "
Nov 02 15:17:42.189508 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.189296    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-openshift-ca" pod "d211982c-f8e5-411d-9e60-ef9f6fe95bb2" (UID: "d211982c-f8e5-411d-9e60-ef9f6fe95bb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:42.189508 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.189473    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:42.191042 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.191015    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-xks-ca" pod "d211982c-f8e5-411d-9e60-ef9f6fe95bb2" (UID: "d211982c-f8e5-411d-9e60-ef9f6fe95bb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:42.191042 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.191025    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-kube-api-access-xcs5p" pod "d211982c-f8e5-411d-9e60-ef9f6fe95bb2" (UID: "d211982c-f8e5-411d-9e60-ef9f6fe95bb2"). InnerVolumeSpecName "kube-api-access-xcs5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:42.290442 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.290407    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:42.290442 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.290440    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcs5p\" (UniqueName: \"kubernetes.io/projected/d211982c-f8e5-411d-9e60-ef9f6fe95bb2-kube-api-access-xcs5p\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:42.732542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.732505    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 15:17:42.732930 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.732569    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" podUID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 15:17:42.901652 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.901574    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"
Nov 02 15:17:42.901808 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.901578    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z" event={"ID":"d211982c-f8e5-411d-9e60-ef9f6fe95bb2","Type":"ContainerDied","Data":"97bba7069728ca68275df77cef039b336a229dc87d119d1bc4e3e0d7a52fce18"}
Nov 02 15:17:42.901808 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:42.901700    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97bba7069728ca68275df77cef039b336a229dc87d119d1bc4e3e0d7a52fce18"
Nov 02 15:17:43.091034 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.091002    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"]
Nov 02 15:17:43.091751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.091727    4891 cpu_manager.go:414] "containerMap: removing container" podUID="d211982c-f8e5-411d-9e60-ef9f6fe95bb2" containerName="revoke-keys"
Nov 02 15:17:43.091751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.091749    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d211982c-f8e5-411d-9e60-ef9f6fe95bb2" containerName="revoke-keys"
Nov 02 15:17:43.091944 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.091827    4891 cpu_manager.go:414] "containerMap: removing container" podUID="2ef8e0fb-be13-408f-b2f7-69c70e569e0d" containerName="revoke-keys"
Nov 02 15:17:43.091944 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.091836    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef8e0fb-be13-408f-b2f7-69c70e569e0d" containerName="revoke-keys"
Nov 02 15:17:43.092047 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.091980    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ef8e0fb-be13-408f-b2f7-69c70e569e0d" containerName="revoke-keys"
Nov 02 15:17:43.092047 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.091998    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="d211982c-f8e5-411d-9e60-ef9f6fe95bb2" containerName="revoke-keys"
Nov 02 15:17:43.095915 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.095894    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:43.176384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.175740    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:17:43.176542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.176401    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:17:43.176542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.176512    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:17:43.176662 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.176563    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:17:43.182859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.182829    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"]
Nov 02 15:17:43.202961 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.202922    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52fe2649-gbd2v\" (UID: \"39f838ec-13e3-4c0e-befa-d432b8294837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:43.203127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.203104    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8tv\" (UniqueName: \"kubernetes.io/projected/39f838ec-13e3-4c0e-befa-d432b8294837-kube-api-access-gt8tv\") pod \"maas-api-revoke-sub-52fe2649-gbd2v\" (UID: \"39f838ec-13e3-4c0e-befa-d432b8294837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:43.203294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.203280    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52fe2649-gbd2v\" (UID: \"39f838ec-13e3-4c0e-befa-d432b8294837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:43.304581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.304540    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8tv\" (UniqueName: \"kubernetes.io/projected/39f838ec-13e3-4c0e-befa-d432b8294837-kube-api-access-gt8tv\") pod \"maas-api-revoke-sub-52fe2649-gbd2v\" (UID: \"39f838ec-13e3-4c0e-befa-d432b8294837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:43.304759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.304689    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52fe2649-gbd2v\" (UID: \"39f838ec-13e3-4c0e-befa-d432b8294837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:43.304759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.304730    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52fe2649-gbd2v\" (UID: \"39f838ec-13e3-4c0e-befa-d432b8294837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:43.305418 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.305379    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52fe2649-gbd2v\" (UID: \"39f838ec-13e3-4c0e-befa-d432b8294837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:43.307103 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.307084    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52fe2649-gbd2v\" (UID: \"39f838ec-13e3-4c0e-befa-d432b8294837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:43.372077 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.372036    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8tv\" (UniqueName: \"kubernetes.io/projected/39f838ec-13e3-4c0e-befa-d432b8294837-kube-api-access-gt8tv\") pod \"maas-api-revoke-sub-52fe2649-gbd2v\" (UID: \"39f838ec-13e3-4c0e-befa-d432b8294837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:43.419142 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.419112    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:43.732502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.732416    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 15:17:43.732502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.732472    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" podUID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 15:17:43.830856 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:43.830818    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f838ec_13e3_4c0e_befa_d432b8294837.slice/crio-cc531467227d68eca3cbddd3da927da7bc4e34f280a4332585a1456ba76d2af8 WatchSource:0}: Error finding container cc531467227d68eca3cbddd3da927da7bc4e34f280a4332585a1456ba76d2af8: Status 404 returned error can't find the container with id cc531467227d68eca3cbddd3da927da7bc4e34f280a4332585a1456ba76d2af8
Nov 02 15:17:43.831466 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.831431    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"]
Nov 02 15:17:43.912121 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:43.912082    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v" event={"ID":"39f838ec-13e3-4c0e-befa-d432b8294837","Type":"ContainerStarted","Data":"cc531467227d68eca3cbddd3da927da7bc4e34f280a4332585a1456ba76d2af8"}
Nov 02 15:17:44.298964 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:17:44.298933    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:17:44.732214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:44.732176    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 15:17:44.732431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:44.732232    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" podUID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 15:17:44.921517 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:44.921480    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v" event={"ID":"39f838ec-13e3-4c0e-befa-d432b8294837","Type":"ContainerStarted","Data":"ccb9b3aa14c666b54d02f96fa02e1342745bc158ef3101aa19f1e4736308679f"}
Nov 02 15:17:45.043238 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:45.043176    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v" podStartSLOduration=2.043125632 podStartE2EDuration="2.043125632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:45.042448627 +0000 UTC m=+1272.635065437" watchObservedRunningTime="2026-11-02 15:17:45.043125632 +0000 UTC m=+1272.635742435"
Nov 02 15:17:45.098771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:45.098728    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"]
Nov 02 15:17:45.103271 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:45.103246    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"]
Nov 02 15:17:45.107441 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:45.107416    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-405f50c3-z29ft"]
Nov 02 15:17:45.111548 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:45.111525    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d50c9d3-k5n26"]
Nov 02 15:17:45.732306 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:45.732272    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 15:17:45.732531 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:45.732356    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" podUID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 15:17:45.932978 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:45.932949    4891 generic.go:356] "Generic (PLEG): container finished" podID="39f838ec-13e3-4c0e-befa-d432b8294837" containerID="ccb9b3aa14c666b54d02f96fa02e1342745bc158ef3101aa19f1e4736308679f" exitCode=0
Nov 02 15:17:45.933362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:45.933039    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v" event={"ID":"39f838ec-13e3-4c0e-befa-d432b8294837","Type":"ContainerDied","Data":"ccb9b3aa14c666b54d02f96fa02e1342745bc158ef3101aa19f1e4736308679f"}
Nov 02 15:17:46.524294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:46.524261    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e39bd5c-e83e-4c72-a399-ca15e5934a61" path="/var/lib/kubelet/pods/0e39bd5c-e83e-4c72-a399-ca15e5934a61/volumes"
Nov 02 15:17:46.524836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:46.524820    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c01829a-ccdd-44d8-8187-fd36f8d25efa" path="/var/lib/kubelet/pods/4c01829a-ccdd-44d8-8187-fd36f8d25efa/volumes"
Nov 02 15:17:46.732801 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:46.732769    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 15:17:46.732954 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:46.732833    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" podUID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 15:17:47.139358 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.139332    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:47.263855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.263780    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-xks-ca\") pod \"39f838ec-13e3-4c0e-befa-d432b8294837\" (UID: \"39f838ec-13e3-4c0e-befa-d432b8294837\") "
Nov 02 15:17:47.263990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.263894    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-openshift-ca\") pod \"39f838ec-13e3-4c0e-befa-d432b8294837\" (UID: \"39f838ec-13e3-4c0e-befa-d432b8294837\") "
Nov 02 15:17:47.264131 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.264114    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f838ec-13e3-4c0e-befa-d432b8294837-kube-api-access-gt8tv\" (UniqueName: \"kubernetes.io/projected/39f838ec-13e3-4c0e-befa-d432b8294837-kube-api-access-gt8tv\") pod \"39f838ec-13e3-4c0e-befa-d432b8294837\" (UID: \"39f838ec-13e3-4c0e-befa-d432b8294837\") "
Nov 02 15:17:47.265013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.264981    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-openshift-ca" pod "39f838ec-13e3-4c0e-befa-d432b8294837" (UID: "39f838ec-13e3-4c0e-befa-d432b8294837"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:47.266412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.266385    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-xks-ca" pod "39f838ec-13e3-4c0e-befa-d432b8294837" (UID: "39f838ec-13e3-4c0e-befa-d432b8294837"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:47.267133 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.267110    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f838ec-13e3-4c0e-befa-d432b8294837-kube-api-access-gt8tv" pod "39f838ec-13e3-4c0e-befa-d432b8294837" (UID: "39f838ec-13e3-4c0e-befa-d432b8294837"). InnerVolumeSpecName "kube-api-access-gt8tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:47.366272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.366227    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:47.366272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.366274    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8tv\" (UniqueName: \"kubernetes.io/projected/39f838ec-13e3-4c0e-befa-d432b8294837-kube-api-access-gt8tv\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:47.366501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.366293    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f838ec-13e3-4c0e-befa-d432b8294837-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:47.893244 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.893209    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:47.893957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.893934    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:47.894478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.894455    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:17:47.956719 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.956689    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"
Nov 02 15:17:47.956863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.956721    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v" event={"ID":"39f838ec-13e3-4c0e-befa-d432b8294837","Type":"ContainerDied","Data":"cc531467227d68eca3cbddd3da927da7bc4e34f280a4332585a1456ba76d2af8"}
Nov 02 15:17:47.956863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:47.956767    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc531467227d68eca3cbddd3da927da7bc4e34f280a4332585a1456ba76d2af8"
Nov 02 15:17:48.445724 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.445683    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"]
Nov 02 15:17:48.446283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.446269    4891 cpu_manager.go:414] "containerMap: removing container" podUID="39f838ec-13e3-4c0e-befa-d432b8294837" containerName="revoke-keys"
Nov 02 15:17:48.446338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.446283    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f838ec-13e3-4c0e-befa-d432b8294837" containerName="revoke-keys"
Nov 02 15:17:48.446454 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.446442    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f838ec-13e3-4c0e-befa-d432b8294837" containerName="revoke-keys"
Nov 02 15:17:48.450176 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.450156    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:48.502627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.502597    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:17:48.502627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.502597    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:17:48.504143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.504103    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:17:48.504280 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.504230    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:17:48.508920 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.508889    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"]
Nov 02 15:17:48.582266 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.582229    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fdgn\" (UniqueName: \"kubernetes.io/projected/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-kube-api-access-6fdgn\") pod \"maas-api-revoke-sub-13f67bfb-hccq7\" (UID: \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:48.582477 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.582457    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13f67bfb-hccq7\" (UID: \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:48.582688 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.582668    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13f67bfb-hccq7\" (UID: \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:48.683604 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.683573    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13f67bfb-hccq7\" (UID: \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:48.683777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.683694    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fdgn\" (UniqueName: \"kubernetes.io/projected/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-kube-api-access-6fdgn\") pod \"maas-api-revoke-sub-13f67bfb-hccq7\" (UID: \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:48.683838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.683826    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13f67bfb-hccq7\" (UID: \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:48.684302 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.684280    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13f67bfb-hccq7\" (UID: \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:48.686226 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.686202    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13f67bfb-hccq7\" (UID: \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:48.757010 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.756938    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fdgn\" (UniqueName: \"kubernetes.io/projected/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-kube-api-access-6fdgn\") pod \"maas-api-revoke-sub-13f67bfb-hccq7\" (UID: \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:48.770677 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:48.770650    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:49.156080 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:49.156044    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffe0cb7_14d1_4b00_bb3b_f81ba823e8cd.slice/crio-a5e9304ee5a17ef3126f17288af6fe2c7fa94fb44ff99bbc16431293342914d7 WatchSource:0}: Error finding container a5e9304ee5a17ef3126f17288af6fe2c7fa94fb44ff99bbc16431293342914d7: Status 404 returned error can't find the container with id a5e9304ee5a17ef3126f17288af6fe2c7fa94fb44ff99bbc16431293342914d7
Nov 02 15:17:49.156431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:49.156404    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"]
Nov 02 15:17:49.976544 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:49.976497    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7" event={"ID":"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd","Type":"ContainerStarted","Data":"8ef89077a59fc1098347ec243606b9c570893610d74ceb8b4c1acad76a6787d5"}
Nov 02 15:17:49.977116 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:49.976595    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7" event={"ID":"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd","Type":"ContainerStarted","Data":"a5e9304ee5a17ef3126f17288af6fe2c7fa94fb44ff99bbc16431293342914d7"}
Nov 02 15:17:50.099106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:50.099050    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7" podStartSLOduration=2.099035538 podStartE2EDuration="2.099035538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:50.095074815 +0000 UTC m=+1277.687691614" watchObservedRunningTime="2026-11-02 15:17:50.099035538 +0000 UTC m=+1277.691652336"
Nov 02 15:17:50.151930 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:17:50.151893    4891 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffe0cb7_14d1_4b00_bb3b_f81ba823e8cd.slice/crio-conmon-8ef89077a59fc1098347ec243606b9c570893610d74ceb8b4c1acad76a6787d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffe0cb7_14d1_4b00_bb3b_f81ba823e8cd.slice/crio-8ef89077a59fc1098347ec243606b9c570893610d74ceb8b4c1acad76a6787d5.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:17:50.988739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:50.988706    4891 generic.go:356] "Generic (PLEG): container finished" podID="9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd" containerID="8ef89077a59fc1098347ec243606b9c570893610d74ceb8b4c1acad76a6787d5" exitCode=0
Nov 02 15:17:50.989206 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:50.988793    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7" event={"ID":"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd","Type":"ContainerDied","Data":"8ef89077a59fc1098347ec243606b9c570893610d74ceb8b4c1acad76a6787d5"}
Nov 02 15:17:51.350758 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.350725    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"]
Nov 02 15:17:51.356138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.356116    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.409641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.409614    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-42555d-openshift-default-dockercfg-4gl7w\""
Nov 02 15:17:51.416124 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.416093    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"]
Nov 02 15:17:51.424002 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.423972    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.424147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.424039    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04eab7d2-8782-4353-b7b3-1ea5df096979-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.424223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.424144    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxw7\" (UniqueName: \"kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-kube-api-access-rpxw7\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.424283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.424267    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.424471 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.424451    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.424545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.424504    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.424661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.424640    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.424739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.424720    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.424797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.424762    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.526297 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.526264    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.526495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.526361    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.526495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.526399    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.526495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.526434    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.526495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.526478    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.526730 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.526500    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.526730 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.526532    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.527498 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.527172    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04eab7d2-8782-4353-b7b3-1ea5df096979-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.527498 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.527267    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxw7\" (UniqueName: \"kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-kube-api-access-rpxw7\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.527498 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.527468    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.527751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.527529    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.527751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.527708    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.527974 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.527954    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.528043 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.528002    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04eab7d2-8782-4353-b7b3-1ea5df096979-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.540183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.540156    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.548803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.548777    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.587254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.587229    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.588231 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.588208    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxw7\" (UniqueName: \"kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-kube-api-access-rpxw7\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:51.669941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:51.669869    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:52.118684 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:52.117634    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04eab7d2_8782_4353_b7b3_1ea5df096979.slice/crio-efe5d80630bd37cb74fc18d1b8f1b13c6562093df575df2c4a518145afa556ed WatchSource:0}: Error finding container efe5d80630bd37cb74fc18d1b8f1b13c6562093df575df2c4a518145afa556ed: Status 404 returned error can't find the container with id efe5d80630bd37cb74fc18d1b8f1b13c6562093df575df2c4a518145afa556ed
Nov 02 15:17:52.122423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.122397    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"]
Nov 02 15:17:52.123654 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.123606    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:52.123767 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.123702    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:52.123767 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.123740    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:52.150249 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.150226    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:52.238199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.238160    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-xks-ca\") pod \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\" (UID: \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\") "
Nov 02 15:17:52.238395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.238347    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-kube-api-access-6fdgn\" (UniqueName: \"kubernetes.io/projected/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-kube-api-access-6fdgn\") pod \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\" (UID: \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\") "
Nov 02 15:17:52.238483 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.238426    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-openshift-ca\") pod \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\" (UID: \"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd\") "
Nov 02 15:17:52.239163 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.239118    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-openshift-ca" pod "9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd" (UID: "9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:52.240882 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.240859    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-kube-api-access-6fdgn" pod "9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd" (UID: "9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd"). InnerVolumeSpecName "kube-api-access-6fdgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:52.240990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.240925    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-xks-ca" pod "9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd" (UID: "9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:52.340904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.340860    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:52.340904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.340899    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fdgn\" (UniqueName: \"kubernetes.io/projected/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-kube-api-access-6fdgn\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:52.341124 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.340915    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:52.597049 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.596960    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"]
Nov 02 15:17:52.597249 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.597222    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerName="istio-proxy" containerID="cri-o://f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7" gracePeriod=30
Nov 02 15:17:52.681865 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.681556    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"]
Nov 02 15:17:52.681865 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:52.681798    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" podUID="13432aa0-5907-4e63-8d12-cacad202b86b" containerName="istio-proxy" containerID="cri-o://993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757" gracePeriod=30
Nov 02 15:17:53.007211 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:53.007131    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"
Nov 02 15:17:53.007211 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:53.007175    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7" event={"ID":"9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd","Type":"ContainerDied","Data":"a5e9304ee5a17ef3126f17288af6fe2c7fa94fb44ff99bbc16431293342914d7"}
Nov 02 15:17:53.007420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:53.007222    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5e9304ee5a17ef3126f17288af6fe2c7fa94fb44ff99bbc16431293342914d7"
Nov 02 15:17:53.010013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:53.009238    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" event={"ID":"04eab7d2-8782-4353-b7b3-1ea5df096979","Type":"ContainerStarted","Data":"84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c"}
Nov 02 15:17:53.010013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:53.009290    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" event={"ID":"04eab7d2-8782-4353-b7b3-1ea5df096979","Type":"ContainerStarted","Data":"efe5d80630bd37cb74fc18d1b8f1b13c6562093df575df2c4a518145afa556ed"}
Nov 02 15:17:53.143567 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:53.143515    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" podStartSLOduration=2.143500406 podStartE2EDuration="2.143500406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:53.140054493 +0000 UTC m=+1280.732671293" watchObservedRunningTime="2026-11-02 15:17:53.143500406 +0000 UTC m=+1280.736117205"
Nov 02 15:17:53.670572 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:53.670533    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:53.672129 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:53.672095    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:17:53.672294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:53.672170    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" podUID="04eab7d2-8782-4353-b7b3-1ea5df096979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 15:17:54.483901 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.483860    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"]
Nov 02 15:17:54.484778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.484748    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd" containerName="revoke-keys"
Nov 02 15:17:54.484778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.484773    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd" containerName="revoke-keys"
Nov 02 15:17:54.484994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.484979    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd" containerName="revoke-keys"
Nov 02 15:17:54.490138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.490118    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:54.564333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.560012    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"]
Nov 02 15:17:54.591382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.588441    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:17:54.591382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.588469    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:17:54.591382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.588471    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:17:54.592125 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.592102    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:17:54.598889 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.598866    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rrlb\" (UniqueName: \"kubernetes.io/projected/c2384dc8-971f-4170-9779-a2db56e05436-kube-api-access-7rrlb\") pod \"maas-api-revoke-sub-1913046a-8sm5g\" (UID: \"c2384dc8-971f-4170-9779-a2db56e05436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:54.599001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.598983    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1913046a-8sm5g\" (UID: \"c2384dc8-971f-4170-9779-a2db56e05436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:54.599052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.599035    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1913046a-8sm5g\" (UID: \"c2384dc8-971f-4170-9779-a2db56e05436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:54.670824 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.670786    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:17:54.671063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.670853    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" podUID="04eab7d2-8782-4353-b7b3-1ea5df096979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 15:17:54.700155 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.700120    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrlb\" (UniqueName: \"kubernetes.io/projected/c2384dc8-971f-4170-9779-a2db56e05436-kube-api-access-7rrlb\") pod \"maas-api-revoke-sub-1913046a-8sm5g\" (UID: \"c2384dc8-971f-4170-9779-a2db56e05436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:54.700380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.700325    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1913046a-8sm5g\" (UID: \"c2384dc8-971f-4170-9779-a2db56e05436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:54.700454 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.700388    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1913046a-8sm5g\" (UID: \"c2384dc8-971f-4170-9779-a2db56e05436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:54.701132 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.701104    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1913046a-8sm5g\" (UID: \"c2384dc8-971f-4170-9779-a2db56e05436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:54.702912 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.702887    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1913046a-8sm5g\" (UID: \"c2384dc8-971f-4170-9779-a2db56e05436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:54.769986 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.769953    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rrlb\" (UniqueName: \"kubernetes.io/projected/c2384dc8-971f-4170-9779-a2db56e05436-kube-api-access-7rrlb\") pod \"maas-api-revoke-sub-1913046a-8sm5g\" (UID: \"c2384dc8-971f-4170-9779-a2db56e05436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:54.804580 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:54.804547    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:55.236582 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:55.236551    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2384dc8_971f_4170_9779_a2db56e05436.slice/crio-ab14653a59a302ce0e37ae1c1bf3b9c989ef05b4b31d8bfeb4ff4253ac5ca575 WatchSource:0}: Error finding container ab14653a59a302ce0e37ae1c1bf3b9c989ef05b4b31d8bfeb4ff4253ac5ca575: Status 404 returned error can't find the container with id ab14653a59a302ce0e37ae1c1bf3b9c989ef05b4b31d8bfeb4ff4253ac5ca575
Nov 02 15:17:55.236866 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:55.236848    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"]
Nov 02 15:17:55.671334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:55.670659    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:17:55.671334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:55.670714    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" podUID="04eab7d2-8782-4353-b7b3-1ea5df096979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 15:17:56.038532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:56.038492    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g" event={"ID":"c2384dc8-971f-4170-9779-a2db56e05436","Type":"ContainerStarted","Data":"a256977d0a4a3b9858d2e42e6025ee517e58fa3a29945bce852ada11933e3bdd"}
Nov 02 15:17:56.038719 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:56.038563    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g" event={"ID":"c2384dc8-971f-4170-9779-a2db56e05436","Type":"ContainerStarted","Data":"ab14653a59a302ce0e37ae1c1bf3b9c989ef05b4b31d8bfeb4ff4253ac5ca575"}
Nov 02 15:17:56.076149 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:56.076105    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"]
Nov 02 15:17:56.122530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:56.122499    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138d9353-x6fwj"]
Nov 02 15:17:56.184729 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:56.184660    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g" podStartSLOduration=2.184645232 podStartE2EDuration="2.184645232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:56.181218793 +0000 UTC m=+1283.773835618" watchObservedRunningTime="2026-11-02 15:17:56.184645232 +0000 UTC m=+1283.777262034"
Nov 02 15:17:56.523780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:56.523747    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191aab13-55f5-41fa-8c4f-c615ad7f21f6" path="/var/lib/kubelet/pods/191aab13-55f5-41fa-8c4f-c615ad7f21f6/volumes"
Nov 02 15:17:56.670736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:56.670645    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:17:56.670736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:56.670714    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" podUID="04eab7d2-8782-4353-b7b3-1ea5df096979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 15:17:57.072842 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.072809    4891 generic.go:356] "Generic (PLEG): container finished" podID="c2384dc8-971f-4170-9779-a2db56e05436" containerID="a256977d0a4a3b9858d2e42e6025ee517e58fa3a29945bce852ada11933e3bdd" exitCode=0
Nov 02 15:17:57.073373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.072901    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g" event={"ID":"c2384dc8-971f-4170-9779-a2db56e05436","Type":"ContainerDied","Data":"a256977d0a4a3b9858d2e42e6025ee517e58fa3a29945bce852ada11933e3bdd"}
Nov 02 15:17:57.640147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.640114    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"]
Nov 02 15:17:57.662704 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.662667    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:17:57.678651 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.678612    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:17:57.678808 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.678674    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" podUID="04eab7d2-8782-4353-b7b3-1ea5df096979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 15:17:57.702256 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.702227    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"]
Nov 02 15:17:57.775721 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.775679    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4hhj\" (UniqueName: \"kubernetes.io/projected/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-kube-api-access-r4hhj\") pod \"maas-api-revoke-sub-ec5ffb06-lh4bk\" (UID: \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:17:57.775948 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.775731    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec5ffb06-lh4bk\" (UID: \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:17:57.776011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.775943    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec5ffb06-lh4bk\" (UID: \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:17:57.878382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.878154    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4hhj\" (UniqueName: \"kubernetes.io/projected/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-kube-api-access-r4hhj\") pod \"maas-api-revoke-sub-ec5ffb06-lh4bk\" (UID: \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:17:57.878382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.878220    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec5ffb06-lh4bk\" (UID: \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:17:57.878613 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.878495    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec5ffb06-lh4bk\" (UID: \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:17:57.880355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.880097    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec5ffb06-lh4bk\" (UID: \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:17:57.881569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.881500    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec5ffb06-lh4bk\" (UID: \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:17:57.965421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.965063    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:17:57.983418 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:57.983004    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4hhj\" (UniqueName: \"kubernetes.io/projected/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-kube-api-access-r4hhj\") pod \"maas-api-revoke-sub-ec5ffb06-lh4bk\" (UID: \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:17:58.008077 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.008048    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:17:58.035495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.035470    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:17:58.083028 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.082992    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce3c572-758e-4a42-92e1-8f10bee50a99-istiod-ca-cert" pod "1ce3c572-758e-4a42-92e1-8f10bee50a99" (UID: "1ce3c572-758e-4a42-92e1-8f10bee50a99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:58.092572 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.082304    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce3c572-758e-4a42-92e1-8f10bee50a99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3c572-758e-4a42-92e1-8f10bee50a99-istiod-ca-cert\") pod \"1ce3c572-758e-4a42-92e1-8f10bee50a99\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") "
Nov 02 15:17:58.093923 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.090680    4891 generic.go:356] "Generic (PLEG): container finished" podID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerID="f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7" exitCode=0
Nov 02 15:17:58.093923 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.090717    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" event={"ID":"1ce3c572-758e-4a42-92e1-8f10bee50a99","Type":"ContainerDied","Data":"f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7"}
Nov 02 15:17:58.093923 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.092874    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" event={"ID":"1ce3c572-758e-4a42-92e1-8f10bee50a99","Type":"ContainerDied","Data":"84acb7b8ba54a9c35f5c4b1b36b0af1db830abdfb019bcbf1bf617ee69a73acf"}
Nov 02 15:17:58.093923 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.092910    4891 scope.go:122] "RemoveContainer" containerID="f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7"
Nov 02 15:17:58.093923 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.090787    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"
Nov 02 15:17:58.101940 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.101602    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13432aa0-5907-4e63-8d12-cacad202b86b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13432aa0-5907-4e63-8d12-cacad202b86b-istiod-ca-cert\") pod \"13432aa0-5907-4e63-8d12-cacad202b86b\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") "
Nov 02 15:17:58.101940 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.101664    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-certs\") pod \"1ce3c572-758e-4a42-92e1-8f10bee50a99\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") "
Nov 02 15:17:58.101940 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.101744    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-kube-api-access-bxcmb\" (UniqueName: \"kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-kube-api-access-bxcmb\") pod \"13432aa0-5907-4e63-8d12-cacad202b86b\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") "
Nov 02 15:17:58.101940 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.101781    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-credential-socket\") pod \"13432aa0-5907-4e63-8d12-cacad202b86b\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") "
Nov 02 15:17:58.101940 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.101815    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-socket\") pod \"13432aa0-5907-4e63-8d12-cacad202b86b\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") "
Nov 02 15:17:58.102278 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.102116    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-certs" pod "1ce3c572-758e-4a42-92e1-8f10bee50a99" (UID: "1ce3c572-758e-4a42-92e1-8f10bee50a99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:58.102868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.101844    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-token\" (UniqueName: \"kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-token\") pod \"1ce3c572-758e-4a42-92e1-8f10bee50a99\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") "
Nov 02 15:17:58.102868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.102429    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-envoy\") pod \"1ce3c572-758e-4a42-92e1-8f10bee50a99\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") "
Nov 02 15:17:58.102868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.102494    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13432aa0-5907-4e63-8d12-cacad202b86b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13432aa0-5907-4e63-8d12-cacad202b86b-istio-podinfo\") pod \"13432aa0-5907-4e63-8d12-cacad202b86b\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") "
Nov 02 15:17:58.102868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.102523    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13432aa0-5907-4e63-8d12-cacad202b86b-istiod-ca-cert" pod "13432aa0-5907-4e63-8d12-cacad202b86b" (UID: "13432aa0-5907-4e63-8d12-cacad202b86b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:58.102868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.102614    4891 generic.go:356] "Generic (PLEG): container finished" podID="13432aa0-5907-4e63-8d12-cacad202b86b" containerID="993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757" exitCode=0
Nov 02 15:17:58.102868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.102637    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-data\") pod \"13432aa0-5907-4e63-8d12-cacad202b86b\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") "
Nov 02 15:17:58.102868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.102721    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-envoy\") pod \"13432aa0-5907-4e63-8d12-cacad202b86b\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") "
Nov 02 15:17:58.102868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.102752    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-certs\") pod \"13432aa0-5907-4e63-8d12-cacad202b86b\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") "
Nov 02 15:17:58.102868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.102805    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-credential-socket\") pod \"1ce3c572-758e-4a42-92e1-8f10bee50a99\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") "
Nov 02 15:17:58.102868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.102834    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-istio-token\" (UniqueName: \"kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-istio-token\") pod \"13432aa0-5907-4e63-8d12-cacad202b86b\" (UID: \"13432aa0-5907-4e63-8d12-cacad202b86b\") "
Nov 02 15:17:58.102868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.102872    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-kube-api-access-v8mjd\" (UniqueName: \"kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-kube-api-access-v8mjd\") pod \"1ce3c572-758e-4a42-92e1-8f10bee50a99\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") "
Nov 02 15:17:58.103607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.103046    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-podinfo\") pod \"1ce3c572-758e-4a42-92e1-8f10bee50a99\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") "
Nov 02 15:17:58.103607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.103105    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-socket\") pod \"1ce3c572-758e-4a42-92e1-8f10bee50a99\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") "
Nov 02 15:17:58.103607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.103175    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-data\") pod \"1ce3c572-758e-4a42-92e1-8f10bee50a99\" (UID: \"1ce3c572-758e-4a42-92e1-8f10bee50a99\") "
Nov 02 15:17:58.104031 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.103994    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"
Nov 02 15:17:58.104031 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.104015    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3c572-758e-4a42-92e1-8f10bee50a99-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.104269 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.104038    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13432aa0-5907-4e63-8d12-cacad202b86b-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.104269 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.104054    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.104269 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.104135    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" event={"ID":"13432aa0-5907-4e63-8d12-cacad202b86b","Type":"ContainerDied","Data":"993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757"}
Nov 02 15:17:58.104269 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.104166    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548" event={"ID":"13432aa0-5907-4e63-8d12-cacad202b86b","Type":"ContainerDied","Data":"5a0c2ce1b864619440c6a942bbbc3d7d776a4e84d852bef29543fcb43ce65325"}
Nov 02 15:17:58.104572 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.104380    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-data" pod "1ce3c572-758e-4a42-92e1-8f10bee50a99" (UID: "1ce3c572-758e-4a42-92e1-8f10bee50a99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:58.104572 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.104389    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-socket" pod "13432aa0-5907-4e63-8d12-cacad202b86b" (UID: "13432aa0-5907-4e63-8d12-cacad202b86b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:58.104681 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.104591    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-credential-socket" pod "13432aa0-5907-4e63-8d12-cacad202b86b" (UID: "13432aa0-5907-4e63-8d12-cacad202b86b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:58.105233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.105205    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-data" pod "13432aa0-5907-4e63-8d12-cacad202b86b" (UID: "13432aa0-5907-4e63-8d12-cacad202b86b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:58.107572 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.107546    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-certs" pod "13432aa0-5907-4e63-8d12-cacad202b86b" (UID: "13432aa0-5907-4e63-8d12-cacad202b86b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:58.109589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.109403    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-credential-socket" pod "1ce3c572-758e-4a42-92e1-8f10bee50a99" (UID: "1ce3c572-758e-4a42-92e1-8f10bee50a99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:58.109769 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.109726    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-socket" pod "1ce3c572-758e-4a42-92e1-8f10bee50a99" (UID: "1ce3c572-758e-4a42-92e1-8f10bee50a99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:58.112407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.111842    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-envoy" pod "13432aa0-5907-4e63-8d12-cacad202b86b" (UID: "13432aa0-5907-4e63-8d12-cacad202b86b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:58.116914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.116885    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-kube-api-access-bxcmb" pod "13432aa0-5907-4e63-8d12-cacad202b86b" (UID: "13432aa0-5907-4e63-8d12-cacad202b86b"). InnerVolumeSpecName "kube-api-access-bxcmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:58.122984 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.122947    4891 scope.go:122] "RemoveContainer" containerID="f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7"
Nov 02 15:17:58.123096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.122945    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13432aa0-5907-4e63-8d12-cacad202b86b-istio-podinfo" pod "13432aa0-5907-4e63-8d12-cacad202b86b" (UID: "13432aa0-5907-4e63-8d12-cacad202b86b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:58.126034 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:17:58.125811    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7\": container with ID starting with f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7 not found: ID does not exist" containerID="f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7"
Nov 02 15:17:58.126034 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.125853    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7"} err="failed to get container status \"f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7\": rpc error: code = NotFound desc = could not find container \"f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7\": container with ID starting with f88f338194a32f5d5286154c1ece29acd99eca44069aad6f2d3ea2ec05bca1e7 not found: ID does not exist"
Nov 02 15:17:58.126034 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.125879    4891 scope.go:122] "RemoveContainer" containerID="993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757"
Nov 02 15:17:58.134252 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.134198    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-istio-token" pod "13432aa0-5907-4e63-8d12-cacad202b86b" (UID: "13432aa0-5907-4e63-8d12-cacad202b86b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:58.134623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.134584    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-token" pod "1ce3c572-758e-4a42-92e1-8f10bee50a99" (UID: "1ce3c572-758e-4a42-92e1-8f10bee50a99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:58.146090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.146049    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-envoy" pod "1ce3c572-758e-4a42-92e1-8f10bee50a99" (UID: "1ce3c572-758e-4a42-92e1-8f10bee50a99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:58.147494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.147414    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-podinfo" pod "1ce3c572-758e-4a42-92e1-8f10bee50a99" (UID: "1ce3c572-758e-4a42-92e1-8f10bee50a99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:58.148492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.148449    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-kube-api-access-v8mjd" pod "1ce3c572-758e-4a42-92e1-8f10bee50a99" (UID: "1ce3c572-758e-4a42-92e1-8f10bee50a99"). InnerVolumeSpecName "kube-api-access-v8mjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:58.167049 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.166918    4891 scope.go:122] "RemoveContainer" containerID="993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757"
Nov 02 15:17:58.167618 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:17:58.167532    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757\": container with ID starting with 993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757 not found: ID does not exist" containerID="993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757"
Nov 02 15:17:58.167618 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.167573    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757"} err="failed to get container status \"993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757\": rpc error: code = NotFound desc = could not find container \"993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757\": container with ID starting with 993fe46ab10204da0800e240e87137f0b72b36bf3a3ce536d69b977163009757 not found: ID does not exist"
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208626    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxcmb\" (UniqueName: \"kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-kube-api-access-bxcmb\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208663    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208678    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208690    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208703    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208714    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13432aa0-5907-4e63-8d12-cacad202b86b-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208726    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208739    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208750    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13432aa0-5907-4e63-8d12-cacad202b86b-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208762    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208775    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13432aa0-5907-4e63-8d12-cacad202b86b-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208787    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8mjd\" (UniqueName: \"kubernetes.io/projected/1ce3c572-758e-4a42-92e1-8f10bee50a99-kube-api-access-v8mjd\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208799    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208811    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.208845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.208823    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce3c572-758e-4a42-92e1-8f10bee50a99-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.338900 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.338870    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:58.412128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.411957    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-openshift-ca\") pod \"c2384dc8-971f-4170-9779-a2db56e05436\" (UID: \"c2384dc8-971f-4170-9779-a2db56e05436\") "
Nov 02 15:17:58.412212 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.412139    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-xks-ca\") pod \"c2384dc8-971f-4170-9779-a2db56e05436\" (UID: \"c2384dc8-971f-4170-9779-a2db56e05436\") "
Nov 02 15:17:58.412212 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.412181    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2384dc8-971f-4170-9779-a2db56e05436-kube-api-access-7rrlb\" (UniqueName: \"kubernetes.io/projected/c2384dc8-971f-4170-9779-a2db56e05436-kube-api-access-7rrlb\") pod \"c2384dc8-971f-4170-9779-a2db56e05436\" (UID: \"c2384dc8-971f-4170-9779-a2db56e05436\") "
Nov 02 15:17:58.412449 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.412420    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-openshift-ca" pod "c2384dc8-971f-4170-9779-a2db56e05436" (UID: "c2384dc8-971f-4170-9779-a2db56e05436"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:58.412721 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.412704    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.414239 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.414218    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-xks-ca" pod "c2384dc8-971f-4170-9779-a2db56e05436" (UID: "c2384dc8-971f-4170-9779-a2db56e05436"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:58.414524 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.414500    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2384dc8-971f-4170-9779-a2db56e05436-kube-api-access-7rrlb" pod "c2384dc8-971f-4170-9779-a2db56e05436" (UID: "c2384dc8-971f-4170-9779-a2db56e05436"). InnerVolumeSpecName "kube-api-access-7rrlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:58.424434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.424410    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"]
Nov 02 15:17:58.424528 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:58.424494    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb1f524_c2db_4ae8_a4de_6a52efd31d86.slice/crio-53145eaf712aeee6eb4ef2b197f38d1d12f7efd572e59fc6bd0c1a6fdc1b56c6 WatchSource:0}: Error finding container 53145eaf712aeee6eb4ef2b197f38d1d12f7efd572e59fc6bd0c1a6fdc1b56c6: Status 404 returned error can't find the container with id 53145eaf712aeee6eb4ef2b197f38d1d12f7efd572e59fc6bd0c1a6fdc1b56c6
Nov 02 15:17:58.513558 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.513519    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2384dc8-971f-4170-9779-a2db56e05436-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.513558 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.513556    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rrlb\" (UniqueName: \"kubernetes.io/projected/c2384dc8-971f-4170-9779-a2db56e05436-kube-api-access-7rrlb\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:17:58.587641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.587602    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"]
Nov 02 15:17:58.601579 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.601537    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qg548"]
Nov 02 15:17:58.617253 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.617221    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"]
Nov 02 15:17:58.617994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.617961    4891 cpu_manager.go:414] "containerMap: removing container" podUID="13432aa0-5907-4e63-8d12-cacad202b86b" containerName="istio-proxy"
Nov 02 15:17:58.617994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.617983    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13432aa0-5907-4e63-8d12-cacad202b86b" containerName="istio-proxy"
Nov 02 15:17:58.618248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.618009    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerName="istio-proxy"
Nov 02 15:17:58.618248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.618015    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerName="istio-proxy"
Nov 02 15:17:58.618248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.618044    4891 cpu_manager.go:414] "containerMap: removing container" podUID="c2384dc8-971f-4170-9779-a2db56e05436" containerName="revoke-keys"
Nov 02 15:17:58.618248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.618050    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2384dc8-971f-4170-9779-a2db56e05436" containerName="revoke-keys"
Nov 02 15:17:58.618248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.618145    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerName="istio-proxy"
Nov 02 15:17:58.618248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.618154    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2384dc8-971f-4170-9779-a2db56e05436" containerName="revoke-keys"
Nov 02 15:17:58.618248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.618163    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="13432aa0-5907-4e63-8d12-cacad202b86b" containerName="istio-proxy"
Nov 02 15:17:58.623229 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.623209    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:17:58.670520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.670496    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:17:58.670629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.670540    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" podUID="04eab7d2-8782-4353-b7b3-1ea5df096979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 15:17:58.682044 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.682015    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"]
Nov 02 15:17:58.709812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.709779    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"]
Nov 02 15:17:58.716735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.716704    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9018a125-mhmd7\" (UID: \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:17:58.716926 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.716901    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6562t\" (UniqueName: \"kubernetes.io/projected/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-kube-api-access-6562t\") pod \"maas-api-revoke-sub-9018a125-mhmd7\" (UID: \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:17:58.717111 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.717088    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9018a125-mhmd7\" (UID: \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:17:58.720012 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.719988    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9"]
Nov 02 15:17:58.818501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.818457    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6562t\" (UniqueName: \"kubernetes.io/projected/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-kube-api-access-6562t\") pod \"maas-api-revoke-sub-9018a125-mhmd7\" (UID: \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:17:58.818701 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.818620    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9018a125-mhmd7\" (UID: \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:17:58.819043 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.818720    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9018a125-mhmd7\" (UID: \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:17:58.819466 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.819447    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9018a125-mhmd7\" (UID: \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:17:58.821140 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.821116    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9018a125-mhmd7\" (UID: \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:17:58.884752 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.884723    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6562t\" (UniqueName: \"kubernetes.io/projected/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-kube-api-access-6562t\") pod \"maas-api-revoke-sub-9018a125-mhmd7\" (UID: \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:17:58.911703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.911669    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-x44q9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:17:58.911858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.911725    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-x44q9" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:58.973688 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.973611    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"]
Nov 02 15:17:58.976664 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.976641    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:17:58.977851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:58.977837    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:17:59.021480 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.021448    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c329fb-a342-4969-9858-54636369dd86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63193799-pt42x\" (UID: \"d8c329fb-a342-4969-9858-54636369dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:17:59.021717 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.021694    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c329fb-a342-4969-9858-54636369dd86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63193799-pt42x\" (UID: \"d8c329fb-a342-4969-9858-54636369dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:17:59.021792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.021738    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lvsc\" (UniqueName: \"kubernetes.io/projected/d8c329fb-a342-4969-9858-54636369dd86-kube-api-access-7lvsc\") pod \"maas-api-revoke-sub-63193799-pt42x\" (UID: \"d8c329fb-a342-4969-9858-54636369dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:17:59.037163 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.037129    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"]
Nov 02 15:17:59.120209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.119962    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g" event={"ID":"c2384dc8-971f-4170-9779-a2db56e05436","Type":"ContainerDied","Data":"ab14653a59a302ce0e37ae1c1bf3b9c989ef05b4b31d8bfeb4ff4253ac5ca575"}
Nov 02 15:17:59.120209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.120048    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab14653a59a302ce0e37ae1c1bf3b9c989ef05b4b31d8bfeb4ff4253ac5ca575"
Nov 02 15:17:59.120209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.120118    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"
Nov 02 15:17:59.124218 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.124179    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c329fb-a342-4969-9858-54636369dd86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63193799-pt42x\" (UID: \"d8c329fb-a342-4969-9858-54636369dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:17:59.125460 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.125150    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c329fb-a342-4969-9858-54636369dd86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63193799-pt42x\" (UID: \"d8c329fb-a342-4969-9858-54636369dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:17:59.126078 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.125753    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lvsc\" (UniqueName: \"kubernetes.io/projected/d8c329fb-a342-4969-9858-54636369dd86-kube-api-access-7lvsc\") pod \"maas-api-revoke-sub-63193799-pt42x\" (UID: \"d8c329fb-a342-4969-9858-54636369dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:17:59.126388 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.126371    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c329fb-a342-4969-9858-54636369dd86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63193799-pt42x\" (UID: \"d8c329fb-a342-4969-9858-54636369dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:17:59.130569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.130546    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c329fb-a342-4969-9858-54636369dd86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63193799-pt42x\" (UID: \"d8c329fb-a342-4969-9858-54636369dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:17:59.131789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.131730    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk" event={"ID":"ebb1f524-c2db-4ae8-a4de-6a52efd31d86","Type":"ContainerStarted","Data":"cad94c142c3403d0dc32949e602b6d228fc6aca6b97c1be7d2d72076a49962f4"}
Nov 02 15:17:59.131789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.131787    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk" event={"ID":"ebb1f524-c2db-4ae8-a4de-6a52efd31d86","Type":"ContainerStarted","Data":"53145eaf712aeee6eb4ef2b197f38d1d12f7efd572e59fc6bd0c1a6fdc1b56c6"}
Nov 02 15:17:59.186990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.186958    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lvsc\" (UniqueName: \"kubernetes.io/projected/d8c329fb-a342-4969-9858-54636369dd86-kube-api-access-7lvsc\") pod \"maas-api-revoke-sub-63193799-pt42x\" (UID: \"d8c329fb-a342-4969-9858-54636369dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:17:59.293604 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.293570    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:17:59.335264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.335192    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk" podStartSLOduration=2.335170735 podStartE2EDuration="2.335170735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:59.331276062 +0000 UTC m=+1286.923892862" watchObservedRunningTime="2026-11-02 15:17:59.335170735 +0000 UTC m=+1286.927787537"
Nov 02 15:17:59.403438 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.403404    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"]
Nov 02 15:17:59.404979 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:17:59.404944    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fff663b_d37d_4b6d_b5a1_2dd0c713421f.slice/crio-90a00ec4b7bb222c008893d16093296ed44c6867d5027621eb33effe56f4ffcb WatchSource:0}: Error finding container 90a00ec4b7bb222c008893d16093296ed44c6867d5027621eb33effe56f4ffcb: Status 404 returned error can't find the container with id 90a00ec4b7bb222c008893d16093296ed44c6867d5027621eb33effe56f4ffcb
Nov 02 15:17:59.674306 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.674279    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:59.674712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.674691    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:59.675531 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.675514    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:17:59.691177 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:17:59.691148    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"]
Nov 02 15:18:00.146300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:00.146265    4891 generic.go:356] "Generic (PLEG): container finished" podID="ebb1f524-c2db-4ae8-a4de-6a52efd31d86" containerID="cad94c142c3403d0dc32949e602b6d228fc6aca6b97c1be7d2d72076a49962f4" exitCode=0
Nov 02 15:18:00.146720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:00.146359    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk" event={"ID":"ebb1f524-c2db-4ae8-a4de-6a52efd31d86","Type":"ContainerDied","Data":"cad94c142c3403d0dc32949e602b6d228fc6aca6b97c1be7d2d72076a49962f4"}
Nov 02 15:18:00.148245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:00.148217    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x" event={"ID":"d8c329fb-a342-4969-9858-54636369dd86","Type":"ContainerStarted","Data":"56fb1e633eb15363f6a09c3519fa6d9bc6c66918b24d4369c2c9f5fbeb4fa23d"}
Nov 02 15:18:00.150206 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:00.150177    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7" event={"ID":"3fff663b-d37d-4b6d-b5a1-2dd0c713421f","Type":"ContainerStarted","Data":"628d683ecc36b743d9ad3c9cc9ea7818119a5ce05247eb5dd90ef71447602085"}
Nov 02 15:18:00.150392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:00.150227    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7" event={"ID":"3fff663b-d37d-4b6d-b5a1-2dd0c713421f","Type":"ContainerStarted","Data":"90a00ec4b7bb222c008893d16093296ed44c6867d5027621eb33effe56f4ffcb"}
Nov 02 15:18:00.396441 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:00.396387    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7" podStartSLOduration=2.396373171 podStartE2EDuration="2.396373171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:00.39295908 +0000 UTC m=+1287.985575880" watchObservedRunningTime="2026-11-02 15:18:00.396373171 +0000 UTC m=+1287.988989970"
Nov 02 15:18:00.526122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:00.526090    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13432aa0-5907-4e63-8d12-cacad202b86b" path="/var/lib/kubelet/pods/13432aa0-5907-4e63-8d12-cacad202b86b/volumes"
Nov 02 15:18:00.526691 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:00.526675    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce3c572-758e-4a42-92e1-8f10bee50a99" path="/var/lib/kubelet/pods/1ce3c572-758e-4a42-92e1-8f10bee50a99/volumes"
Nov 02 15:18:01.169374 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.169342    4891 generic.go:356] "Generic (PLEG): container finished" podID="d8c329fb-a342-4969-9858-54636369dd86" containerID="bd19a1490f1f1c789a613e9ed781d11cdf76eb6001d66de59faf7556bb91f53c" exitCode=0
Nov 02 15:18:01.169859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.169431    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x" event={"ID":"d8c329fb-a342-4969-9858-54636369dd86","Type":"ContainerDied","Data":"bd19a1490f1f1c789a613e9ed781d11cdf76eb6001d66de59faf7556bb91f53c"}
Nov 02 15:18:01.171247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.171221    4891 generic.go:356] "Generic (PLEG): container finished" podID="3fff663b-d37d-4b6d-b5a1-2dd0c713421f" containerID="628d683ecc36b743d9ad3c9cc9ea7818119a5ce05247eb5dd90ef71447602085" exitCode=0
Nov 02 15:18:01.171363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.171295    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7" event={"ID":"3fff663b-d37d-4b6d-b5a1-2dd0c713421f","Type":"ContainerDied","Data":"628d683ecc36b743d9ad3c9cc9ea7818119a5ce05247eb5dd90ef71447602085"}
Nov 02 15:18:01.342776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.342745    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:18:01.458626 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.458540    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-kube-api-access-r4hhj\" (UniqueName: \"kubernetes.io/projected/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-kube-api-access-r4hhj\") pod \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\" (UID: \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\") "
Nov 02 15:18:01.458799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.458713    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-openshift-ca\") pod \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\" (UID: \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\") "
Nov 02 15:18:01.458878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.458797    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-xks-ca\") pod \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\" (UID: \"ebb1f524-c2db-4ae8-a4de-6a52efd31d86\") "
Nov 02 15:18:01.459189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.459142    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-openshift-ca" pod "ebb1f524-c2db-4ae8-a4de-6a52efd31d86" (UID: "ebb1f524-c2db-4ae8-a4de-6a52efd31d86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:01.459370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.459344    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:01.460870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.460838    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-kube-api-access-r4hhj" pod "ebb1f524-c2db-4ae8-a4de-6a52efd31d86" (UID: "ebb1f524-c2db-4ae8-a4de-6a52efd31d86"). InnerVolumeSpecName "kube-api-access-r4hhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:01.460964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.460849    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-xks-ca" pod "ebb1f524-c2db-4ae8-a4de-6a52efd31d86" (UID: "ebb1f524-c2db-4ae8-a4de-6a52efd31d86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:01.560715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.560683    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:01.560715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:01.560712    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4hhj\" (UniqueName: \"kubernetes.io/projected/ebb1f524-c2db-4ae8-a4de-6a52efd31d86-kube-api-access-r4hhj\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:02.181032 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.180994    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk" event={"ID":"ebb1f524-c2db-4ae8-a4de-6a52efd31d86","Type":"ContainerDied","Data":"53145eaf712aeee6eb4ef2b197f38d1d12f7efd572e59fc6bd0c1a6fdc1b56c6"}
Nov 02 15:18:02.181506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.181047    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53145eaf712aeee6eb4ef2b197f38d1d12f7efd572e59fc6bd0c1a6fdc1b56c6"
Nov 02 15:18:02.181506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.181244    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"
Nov 02 15:18:02.401058 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.401033    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:18:02.403812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.403791    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:18:02.475478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.475393    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-kube-api-access-6562t\" (UniqueName: \"kubernetes.io/projected/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-kube-api-access-6562t\") pod \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\" (UID: \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\") "
Nov 02 15:18:02.475478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.475432    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c329fb-a342-4969-9858-54636369dd86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c329fb-a342-4969-9858-54636369dd86-maas-api-xks-ca\") pod \"d8c329fb-a342-4969-9858-54636369dd86\" (UID: \"d8c329fb-a342-4969-9858-54636369dd86\") "
Nov 02 15:18:02.475478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.475459    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c329fb-a342-4969-9858-54636369dd86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c329fb-a342-4969-9858-54636369dd86-maas-api-openshift-ca\") pod \"d8c329fb-a342-4969-9858-54636369dd86\" (UID: \"d8c329fb-a342-4969-9858-54636369dd86\") "
Nov 02 15:18:02.475747 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.475514    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-xks-ca\") pod \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\" (UID: \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\") "
Nov 02 15:18:02.475747 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.475674    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c329fb-a342-4969-9858-54636369dd86-kube-api-access-7lvsc\" (UniqueName: \"kubernetes.io/projected/d8c329fb-a342-4969-9858-54636369dd86-kube-api-access-7lvsc\") pod \"d8c329fb-a342-4969-9858-54636369dd86\" (UID: \"d8c329fb-a342-4969-9858-54636369dd86\") "
Nov 02 15:18:02.475747 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.475720    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-openshift-ca\") pod \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\" (UID: \"3fff663b-d37d-4b6d-b5a1-2dd0c713421f\") "
Nov 02 15:18:02.475966 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.475890    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c329fb-a342-4969-9858-54636369dd86-maas-api-openshift-ca" pod "d8c329fb-a342-4969-9858-54636369dd86" (UID: "d8c329fb-a342-4969-9858-54636369dd86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:02.476435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.476405    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-openshift-ca" pod "3fff663b-d37d-4b6d-b5a1-2dd0c713421f" (UID: "3fff663b-d37d-4b6d-b5a1-2dd0c713421f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:02.476563 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.476447    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c329fb-a342-4969-9858-54636369dd86-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:02.477891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.477865    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c329fb-a342-4969-9858-54636369dd86-maas-api-xks-ca" pod "d8c329fb-a342-4969-9858-54636369dd86" (UID: "d8c329fb-a342-4969-9858-54636369dd86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:02.477891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.477876    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-xks-ca" pod "3fff663b-d37d-4b6d-b5a1-2dd0c713421f" (UID: "3fff663b-d37d-4b6d-b5a1-2dd0c713421f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:02.478124 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.478099    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-kube-api-access-6562t" pod "3fff663b-d37d-4b6d-b5a1-2dd0c713421f" (UID: "3fff663b-d37d-4b6d-b5a1-2dd0c713421f"). InnerVolumeSpecName "kube-api-access-6562t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:02.478221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.478155    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c329fb-a342-4969-9858-54636369dd86-kube-api-access-7lvsc" pod "d8c329fb-a342-4969-9858-54636369dd86" (UID: "d8c329fb-a342-4969-9858-54636369dd86"). InnerVolumeSpecName "kube-api-access-7lvsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:02.578307 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.578278    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lvsc\" (UniqueName: \"kubernetes.io/projected/d8c329fb-a342-4969-9858-54636369dd86-kube-api-access-7lvsc\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:02.578457 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.578304    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:02.578457 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.578346    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6562t\" (UniqueName: \"kubernetes.io/projected/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-kube-api-access-6562t\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:02.578457 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.578355    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c329fb-a342-4969-9858-54636369dd86-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:02.578457 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:02.578365    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fff663b-d37d-4b6d-b5a1-2dd0c713421f-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:03.192521 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:03.192486    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"
Nov 02 15:18:03.192521 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:03.192518    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x" event={"ID":"d8c329fb-a342-4969-9858-54636369dd86","Type":"ContainerDied","Data":"56fb1e633eb15363f6a09c3519fa6d9bc6c66918b24d4369c2c9f5fbeb4fa23d"}
Nov 02 15:18:03.193028 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:03.192570    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56fb1e633eb15363f6a09c3519fa6d9bc6c66918b24d4369c2c9f5fbeb4fa23d"
Nov 02 15:18:03.194063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:03.194038    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"
Nov 02 15:18:03.194186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:03.194084    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7" event={"ID":"3fff663b-d37d-4b6d-b5a1-2dd0c713421f","Type":"ContainerDied","Data":"90a00ec4b7bb222c008893d16093296ed44c6867d5027621eb33effe56f4ffcb"}
Nov 02 15:18:03.194186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:03.194133    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a00ec4b7bb222c008893d16093296ed44c6867d5027621eb33effe56f4ffcb"
Nov 02 15:18:05.253782 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.253739    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"]
Nov 02 15:18:05.254558 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.254540    4891 cpu_manager.go:414] "containerMap: removing container" podUID="d8c329fb-a342-4969-9858-54636369dd86" containerName="revoke-keys"
Nov 02 15:18:05.254620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.254559    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c329fb-a342-4969-9858-54636369dd86" containerName="revoke-keys"
Nov 02 15:18:05.254620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.254613    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ebb1f524-c2db-4ae8-a4de-6a52efd31d86" containerName="revoke-keys"
Nov 02 15:18:05.254725 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.254621    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb1f524-c2db-4ae8-a4de-6a52efd31d86" containerName="revoke-keys"
Nov 02 15:18:05.254725 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.254640    4891 cpu_manager.go:414] "containerMap: removing container" podUID="3fff663b-d37d-4b6d-b5a1-2dd0c713421f" containerName="revoke-keys"
Nov 02 15:18:05.254725 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.254648    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fff663b-d37d-4b6d-b5a1-2dd0c713421f" containerName="revoke-keys"
Nov 02 15:18:05.254864 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.254791    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c329fb-a342-4969-9858-54636369dd86" containerName="revoke-keys"
Nov 02 15:18:05.254864 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.254803    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb1f524-c2db-4ae8-a4de-6a52efd31d86" containerName="revoke-keys"
Nov 02 15:18:05.254864 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.254816    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fff663b-d37d-4b6d-b5a1-2dd0c713421f" containerName="revoke-keys"
Nov 02 15:18:05.259952 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.259930    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:05.312635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.312591    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b47ac47-p7kvm\" (UID: \"3badbb25-fded-43dd-aae7-243e2e0311b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:05.312828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.312643    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b47ac47-p7kvm\" (UID: \"3badbb25-fded-43dd-aae7-243e2e0311b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:05.312828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.312760    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv7ld\" (UniqueName: \"kubernetes.io/projected/3badbb25-fded-43dd-aae7-243e2e0311b0-kube-api-access-zv7ld\") pod \"maas-api-revoke-sub-8b47ac47-p7kvm\" (UID: \"3badbb25-fded-43dd-aae7-243e2e0311b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:05.313682 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.313650    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"]
Nov 02 15:18:05.356755 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.356721    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:18:05.356909 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.356759    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:18:05.356909 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.356842    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:18:05.361226 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.361207    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:18:05.414000 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.413960    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b47ac47-p7kvm\" (UID: \"3badbb25-fded-43dd-aae7-243e2e0311b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:05.414191 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.414008    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b47ac47-p7kvm\" (UID: \"3badbb25-fded-43dd-aae7-243e2e0311b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:05.414191 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.414070    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv7ld\" (UniqueName: \"kubernetes.io/projected/3badbb25-fded-43dd-aae7-243e2e0311b0-kube-api-access-zv7ld\") pod \"maas-api-revoke-sub-8b47ac47-p7kvm\" (UID: \"3badbb25-fded-43dd-aae7-243e2e0311b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:05.414757 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.414733    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b47ac47-p7kvm\" (UID: \"3badbb25-fded-43dd-aae7-243e2e0311b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:05.416520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.416498    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b47ac47-p7kvm\" (UID: \"3badbb25-fded-43dd-aae7-243e2e0311b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:05.477048 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.477007    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv7ld\" (UniqueName: \"kubernetes.io/projected/3badbb25-fded-43dd-aae7-243e2e0311b0-kube-api-access-zv7ld\") pod \"maas-api-revoke-sub-8b47ac47-p7kvm\" (UID: \"3badbb25-fded-43dd-aae7-243e2e0311b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:05.573984 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.573950    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:05.827741 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.827654    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"]
Nov 02 15:18:05.836664 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.836637    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:05.897766 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.897736    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"]
Nov 02 15:18:05.920931 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.920899    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1828c6-62fc-497b-9ef9-e12d001f4683-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:05.921114 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.920940    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:05.921114 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.920965    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkllj\" (UniqueName: \"kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-kube-api-access-dkllj\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:05.921219 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.921105    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:05.921219 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.921143    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:05.921340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.921278    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:05.921436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.921417    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:05.921495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.921480    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:05.921570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.921555    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:05.961987 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:18:05.961957    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3badbb25_fded_43dd_aae7_243e2e0311b0.slice/crio-d87e2e6143d18a11a10d39c7f9f7bcbdef6543599e1a735ab0cd020aca58d7e2 WatchSource:0}: Error finding container d87e2e6143d18a11a10d39c7f9f7bcbdef6543599e1a735ab0cd020aca58d7e2: Status 404 returned error can't find the container with id d87e2e6143d18a11a10d39c7f9f7bcbdef6543599e1a735ab0cd020aca58d7e2
Nov 02 15:18:05.966053 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:05.966027    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"]
Nov 02 15:18:06.022795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.022761    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.022922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.022866    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.022990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.022919    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.022990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.022972    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.023112 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.023083    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1828c6-62fc-497b-9ef9-e12d001f4683-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.023167 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.023125    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.023167 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.023159    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkllj\" (UniqueName: \"kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-kube-api-access-dkllj\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.023270 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.023234    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.023339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.023273    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.023394 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.023356    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-credential-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.023452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.023384    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-socket\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.023580 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.023557    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-data\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.023824 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.023804    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-certs\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.024051 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.024028    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1828c6-62fc-497b-9ef9-e12d001f4683-istiod-ca-cert\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.025568 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.025541    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-envoy\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.025722 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.025707    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-podinfo\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.084265 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.084195    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-token\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.084665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.084645    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkllj\" (UniqueName: \"kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-kube-api-access-dkllj\") pod \"e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") " pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.152971 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.152941    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:06.229957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.229878    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm" event={"ID":"3badbb25-fded-43dd-aae7-243e2e0311b0","Type":"ContainerStarted","Data":"d87e2e6143d18a11a10d39c7f9f7bcbdef6543599e1a735ab0cd020aca58d7e2"}
Nov 02 15:18:06.594745 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:18:06.594710    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1828c6_62fc_497b_9ef9_e12d001f4683.slice/crio-666e092ac947b84f2da4ea97131b86f043e5b0d647982c1dc7057733a07d1986 WatchSource:0}: Error finding container 666e092ac947b84f2da4ea97131b86f043e5b0d647982c1dc7057733a07d1986: Status 404 returned error can't find the container with id 666e092ac947b84f2da4ea97131b86f043e5b0d647982c1dc7057733a07d1986
Nov 02 15:18:06.595724 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.595698    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"]
Nov 02 15:18:06.597395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.597360    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:06.597466 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.597455    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:06.597500 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.597490    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:06.785553 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.785516    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"]
Nov 02 15:18:06.789858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.789840    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:06.866982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.866880    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"]
Nov 02 15:18:06.937510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.937476    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:06.937627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.937535    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:06.937677 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.937653    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:06.937739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.937728    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:06.937914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.937888    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpltz\" (UniqueName: \"kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-kube-api-access-wpltz\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:06.937962 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.937946    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:06.938069 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.938047    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:06.938127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.938100    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:06.938185 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:06.938162    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.039990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.039953    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.040161 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.040060    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.040161 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.040153    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpltz\" (UniqueName: \"kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-kube-api-access-wpltz\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.040341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.040180    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.040341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.040220    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.040341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.040257    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.040341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.040285    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.040538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.040373    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.040538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.040413    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.040538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.040445    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.040753 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.040732    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.040838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.040813    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.040938 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.040914    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.041270 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.041250    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.043059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.043029    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.043207 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.043188    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.053190 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.053160    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"]
Nov 02 15:18:07.058488 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.058467    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8c9a309-8m576"]
Nov 02 15:18:07.101298 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.101269    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.101579 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.101557    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpltz\" (UniqueName: \"kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-kube-api-access-wpltz\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.104620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.104596    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:07.255001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.254957    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" event={"ID":"9d1828c6-62fc-497b-9ef9-e12d001f4683","Type":"ContainerStarted","Data":"0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9"}
Nov 02 15:18:07.255193 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.255063    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" event={"ID":"9d1828c6-62fc-497b-9ef9-e12d001f4683","Type":"ContainerStarted","Data":"666e092ac947b84f2da4ea97131b86f043e5b0d647982c1dc7057733a07d1986"}
Nov 02 15:18:07.257001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.256971    4891 generic.go:356] "Generic (PLEG): container finished" podID="3badbb25-fded-43dd-aae7-243e2e0311b0" containerID="0479f2ce8dc89e852960cf0726ed2b762671e67a4d1c3f87404dab76c6e5e3bd" exitCode=0
Nov 02 15:18:07.257160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.257133    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm" event={"ID":"3badbb25-fded-43dd-aae7-243e2e0311b0","Type":"ContainerDied","Data":"0479f2ce8dc89e852960cf0726ed2b762671e67a4d1c3f87404dab76c6e5e3bd"}
Nov 02 15:18:07.381292 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.381176    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podStartSLOduration=2.3811561230000002 podStartE2EDuration="2.381156123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:07.375065438 +0000 UTC m=+1294.967682237" watchObservedRunningTime="2026-11-02 15:18:07.381156123 +0000 UTC m=+1294.973772926"
Nov 02 15:18:07.498476 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:18:07.498446    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666c6a6f_4079_4d31_b6ff_1f2c1116a0ff.slice/crio-4fa549674da7a9fa2c0d9773cd520ea331e7bf9b2f53bb04f02b664c356aa81a WatchSource:0}: Error finding container 4fa549674da7a9fa2c0d9773cd520ea331e7bf9b2f53bb04f02b664c356aa81a: Status 404 returned error can't find the container with id 4fa549674da7a9fa2c0d9773cd520ea331e7bf9b2f53bb04f02b664c356aa81a
Nov 02 15:18:07.501493 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.501455    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:07.501590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.501524    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:07.501590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.501552    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:07.501651 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:07.501586    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"]
Nov 02 15:18:08.167334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.156505    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:08.167334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.156677    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:08.167334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.156737    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:08.284250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.284215    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" event={"ID":"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff","Type":"ContainerStarted","Data":"a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b"}
Nov 02 15:18:08.284434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.284272    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" event={"ID":"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff","Type":"ContainerStarted","Data":"4fa549674da7a9fa2c0d9773cd520ea331e7bf9b2f53bb04f02b664c356aa81a"}
Nov 02 15:18:08.499445 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.499054    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podStartSLOduration=2.499031986 podStartE2EDuration="2.499031986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:08.497815862 +0000 UTC m=+1296.090432693" watchObservedRunningTime="2026-11-02 15:18:08.499031986 +0000 UTC m=+1296.091648788"
Nov 02 15:18:08.523440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.523411    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8" path="/var/lib/kubelet/pods/ddadd3c9-7301-4ffb-ac4c-8bdc629bccf8/volumes"
Nov 02 15:18:08.534206 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.534185    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:08.710520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.707435    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-xks-ca\") pod \"3badbb25-fded-43dd-aae7-243e2e0311b0\" (UID: \"3badbb25-fded-43dd-aae7-243e2e0311b0\") "
Nov 02 15:18:08.710520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.707690    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-openshift-ca\") pod \"3badbb25-fded-43dd-aae7-243e2e0311b0\" (UID: \"3badbb25-fded-43dd-aae7-243e2e0311b0\") "
Nov 02 15:18:08.710520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.707732    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3badbb25-fded-43dd-aae7-243e2e0311b0-kube-api-access-zv7ld\" (UniqueName: \"kubernetes.io/projected/3badbb25-fded-43dd-aae7-243e2e0311b0-kube-api-access-zv7ld\") pod \"3badbb25-fded-43dd-aae7-243e2e0311b0\" (UID: \"3badbb25-fded-43dd-aae7-243e2e0311b0\") "
Nov 02 15:18:08.710520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.710358    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-openshift-ca" pod "3badbb25-fded-43dd-aae7-243e2e0311b0" (UID: "3badbb25-fded-43dd-aae7-243e2e0311b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:08.712627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.712594    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-xks-ca" pod "3badbb25-fded-43dd-aae7-243e2e0311b0" (UID: "3badbb25-fded-43dd-aae7-243e2e0311b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:08.713606 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.713538    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3badbb25-fded-43dd-aae7-243e2e0311b0-kube-api-access-zv7ld" pod "3badbb25-fded-43dd-aae7-243e2e0311b0" (UID: "3badbb25-fded-43dd-aae7-243e2e0311b0"). InnerVolumeSpecName "kube-api-access-zv7ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:08.809404 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.809373    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:08.809404 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.809401    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv7ld\" (UniqueName: \"kubernetes.io/projected/3badbb25-fded-43dd-aae7-243e2e0311b0-kube-api-access-zv7ld\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:08.809404 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:08.809411    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3badbb25-fded-43dd-aae7-243e2e0311b0-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:09.105799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.105725    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:09.106723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.106700    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:09.106811 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.106747    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:09.171338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.156754    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:09.171338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.156815    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:09.308550 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.308510    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm" event={"ID":"3badbb25-fded-43dd-aae7-243e2e0311b0","Type":"ContainerDied","Data":"d87e2e6143d18a11a10d39c7f9f7bcbdef6543599e1a735ab0cd020aca58d7e2"}
Nov 02 15:18:09.308699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.308573    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d87e2e6143d18a11a10d39c7f9f7bcbdef6543599e1a735ab0cd020aca58d7e2"
Nov 02 15:18:09.308699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.308640    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"
Nov 02 15:18:09.510436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.510358    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"]
Nov 02 15:18:09.511072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.511054    4891 cpu_manager.go:414] "containerMap: removing container" podUID="3badbb25-fded-43dd-aae7-243e2e0311b0" containerName="revoke-keys"
Nov 02 15:18:09.511169 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.511073    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3badbb25-fded-43dd-aae7-243e2e0311b0" containerName="revoke-keys"
Nov 02 15:18:09.511323 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.511295    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="3badbb25-fded-43dd-aae7-243e2e0311b0" containerName="revoke-keys"
Nov 02 15:18:09.515229 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.515210    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:09.520282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.520259    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fa8a91e-cpq28\" (UID: \"307f7aec-bc96-4e78-95db-134ebf285a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:09.520417 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.520327    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fa8a91e-cpq28\" (UID: \"307f7aec-bc96-4e78-95db-134ebf285a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:09.520480 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.520461    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24z9v\" (UniqueName: \"kubernetes.io/projected/307f7aec-bc96-4e78-95db-134ebf285a8e-kube-api-access-24z9v\") pod \"maas-api-revoke-sub-0fa8a91e-cpq28\" (UID: \"307f7aec-bc96-4e78-95db-134ebf285a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:09.634982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.629277    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fa8a91e-cpq28\" (UID: \"307f7aec-bc96-4e78-95db-134ebf285a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:09.634982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.629358    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fa8a91e-cpq28\" (UID: \"307f7aec-bc96-4e78-95db-134ebf285a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:09.634982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.629434    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24z9v\" (UniqueName: \"kubernetes.io/projected/307f7aec-bc96-4e78-95db-134ebf285a8e-kube-api-access-24z9v\") pod \"maas-api-revoke-sub-0fa8a91e-cpq28\" (UID: \"307f7aec-bc96-4e78-95db-134ebf285a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:09.634982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.632919    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:18:09.634982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.633001    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:18:09.634982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.633108    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:18:09.634982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.633187    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:18:09.639757 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.639015    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"]
Nov 02 15:18:09.641793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.641764    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fa8a91e-cpq28\" (UID: \"307f7aec-bc96-4e78-95db-134ebf285a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:09.725019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.724986    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fa8a91e-cpq28\" (UID: \"307f7aec-bc96-4e78-95db-134ebf285a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:09.808787 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.769046    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24z9v\" (UniqueName: \"kubernetes.io/projected/307f7aec-bc96-4e78-95db-134ebf285a8e-kube-api-access-24z9v\") pod \"maas-api-revoke-sub-0fa8a91e-cpq28\" (UID: \"307f7aec-bc96-4e78-95db-134ebf285a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:09.833645 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.833609    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:09.835399 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.835370    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"]
Nov 02 15:18:09.841043 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.841004    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:09.842140 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:09.842111    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"]
Nov 02 15:18:10.037414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.037334    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5fd60c5-8w5m2\" (UID: \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:10.037414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.037384    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5fd60c5-8w5m2\" (UID: \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:10.037651 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.037481    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfc8l\" (UniqueName: \"kubernetes.io/projected/454dc6bb-de62-4247-9127-70bc5d5d0a1e-kube-api-access-cfc8l\") pod \"maas-api-revoke-sub-e5fd60c5-8w5m2\" (UID: \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:10.125365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.112443    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:10.125365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.112497    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:10.139423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.139389    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5fd60c5-8w5m2\" (UID: \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:10.139608 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.139446    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5fd60c5-8w5m2\" (UID: \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:10.139608 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.139492    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfc8l\" (UniqueName: \"kubernetes.io/projected/454dc6bb-de62-4247-9127-70bc5d5d0a1e-kube-api-access-cfc8l\") pod \"maas-api-revoke-sub-e5fd60c5-8w5m2\" (UID: \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:10.140932 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.140902    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5fd60c5-8w5m2\" (UID: \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:10.143983 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.143959    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5fd60c5-8w5m2\" (UID: \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:10.159515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.159405    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:10.159515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.159459    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:10.207603 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.207568    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfc8l\" (UniqueName: \"kubernetes.io/projected/454dc6bb-de62-4247-9127-70bc5d5d0a1e-kube-api-access-cfc8l\") pod \"maas-api-revoke-sub-e5fd60c5-8w5m2\" (UID: \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:10.303016 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.302803    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:10.416633 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:18:10.415302    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307f7aec_bc96_4e78_95db_134ebf285a8e.slice/crio-c80ba7df9ca515aae90c28d2be1183c125274fb265b514ad9ce864022f191b5f WatchSource:0}: Error finding container c80ba7df9ca515aae90c28d2be1183c125274fb265b514ad9ce864022f191b5f: Status 404 returned error can't find the container with id c80ba7df9ca515aae90c28d2be1183c125274fb265b514ad9ce864022f191b5f
Nov 02 15:18:10.420263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.420236    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"]
Nov 02 15:18:10.741876 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:18:10.741851    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454dc6bb_de62_4247_9127_70bc5d5d0a1e.slice/crio-140bfa1b6d0cc7200afda11a0d2a5a6031bbc4544949ac7feb114bc327ba4cb0 WatchSource:0}: Error finding container 140bfa1b6d0cc7200afda11a0d2a5a6031bbc4544949ac7feb114bc327ba4cb0: Status 404 returned error can't find the container with id 140bfa1b6d0cc7200afda11a0d2a5a6031bbc4544949ac7feb114bc327ba4cb0
Nov 02 15:18:10.767413 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:10.755489    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"]
Nov 02 15:18:11.112347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:11.108931    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:11.112347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:11.108993    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:11.163431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:11.163396    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:11.163700 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:11.163671    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:11.369398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:11.362235    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28" event={"ID":"307f7aec-bc96-4e78-95db-134ebf285a8e","Type":"ContainerStarted","Data":"4ac8530de89407dad301a5ae100ca38045c1ccf9ca8ba1431ed53e6f68537695"}
Nov 02 15:18:11.369398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:11.362299    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28" event={"ID":"307f7aec-bc96-4e78-95db-134ebf285a8e","Type":"ContainerStarted","Data":"c80ba7df9ca515aae90c28d2be1183c125274fb265b514ad9ce864022f191b5f"}
Nov 02 15:18:11.369398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:11.366278    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2" event={"ID":"454dc6bb-de62-4247-9127-70bc5d5d0a1e","Type":"ContainerStarted","Data":"b800bc7940d521e2aa4a45f64fff501f916bc8a243b3442c0e5470be188f091f"}
Nov 02 15:18:11.378347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:11.373416    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2" event={"ID":"454dc6bb-de62-4247-9127-70bc5d5d0a1e","Type":"ContainerStarted","Data":"140bfa1b6d0cc7200afda11a0d2a5a6031bbc4544949ac7feb114bc327ba4cb0"}
Nov 02 15:18:11.519433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:11.519363    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28" podStartSLOduration=2.51933699 podStartE2EDuration="2.51933699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:11.492477049 +0000 UTC m=+1299.085093850" watchObservedRunningTime="2026-11-02 15:18:11.51933699 +0000 UTC m=+1299.111953794"
Nov 02 15:18:11.664391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:11.663421    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2" podStartSLOduration=2.663400862 podStartE2EDuration="2.663400862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:11.617708561 +0000 UTC m=+1299.210325364" watchObservedRunningTime="2026-11-02 15:18:11.663400862 +0000 UTC m=+1299.256017662"
Nov 02 15:18:12.113987 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:12.108935    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:12.113987 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:12.108998    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:12.487391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:12.486553    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:12.487391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:12.486615    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:12.550358 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:12.541403    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:12.660391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:12.658303    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:12.735689 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:12.734887    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:12.828334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:12.827719    4891 generic.go:356] "Generic (PLEG): container finished" podID="307f7aec-bc96-4e78-95db-134ebf285a8e" containerID="4ac8530de89407dad301a5ae100ca38045c1ccf9ca8ba1431ed53e6f68537695" exitCode=0
Nov 02 15:18:12.884293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:12.857275    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28" event={"ID":"307f7aec-bc96-4e78-95db-134ebf285a8e","Type":"ContainerDied","Data":"4ac8530de89407dad301a5ae100ca38045c1ccf9ca8ba1431ed53e6f68537695"}
Nov 02 15:18:12.884293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:12.862213    4891 generic.go:356] "Generic (PLEG): container finished" podID="454dc6bb-de62-4247-9127-70bc5d5d0a1e" containerID="b800bc7940d521e2aa4a45f64fff501f916bc8a243b3442c0e5470be188f091f" exitCode=0
Nov 02 15:18:12.884293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:12.862630    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2" event={"ID":"454dc6bb-de62-4247-9127-70bc5d5d0a1e","Type":"ContainerDied","Data":"b800bc7940d521e2aa4a45f64fff501f916bc8a243b3442c0e5470be188f091f"}
Nov 02 15:18:12.894221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:12.893410    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:13.087534 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.087403    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:13.107874 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.107827    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:13.108162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.108125    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:13.160201 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.160163    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:13.160485 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.160456    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:13.495395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.493590    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:13.495395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.494702    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:13.502347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.497800    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:13.502347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.498134    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:13.502347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.498431    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:18:13.502347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.501406    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:13.950994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.865183    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 02 15:18:13.951601 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.865207    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:18:13.960345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.865239    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 02 15:18:13.960801 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.865261    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 15:18:13.964341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.961206    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:18:13.964556 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:13.964536    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:18:14.138332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.131637    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:14.193759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.174547    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:14.221439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.194163    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:18:14.221439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.200039    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:14.221439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.200086    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:14.221439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.203390    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Nov 02 15:18:14.221439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.203520    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:14.221439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.203551    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:14.513391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.512542    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:14.513391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.512966    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:18:14.586336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.545229    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 15:18:14.586336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.577486    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:18:14.650336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.642283    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:18:14.650336 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:18:14.643071    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:18:14.693734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.693689    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:14.712690 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.701241    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:18:14.900336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.897481    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:14.959398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.956453    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:14.981472 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.981422    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 02 15:18:14.983617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.983589    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:18:14.983855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.983841    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:18:14.984085 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:14.984063    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:15.110693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.110658    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:15.110981 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.110938    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:15.151837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.150436    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:15.153155 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.152101    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:18:15.176655 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.175496    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:15.203877 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.176951    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:18:15.210653 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.210618    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:15.210903 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.210876    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:15.318336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.316571    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 15:18:15.321871 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.321493    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 15:18:15.537343 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.528039    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:15.591445 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.591398    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 15:18:15.623455 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.623404    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:18:15.623778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.623745    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:15.717304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.706524    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:15.717304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.707358    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:15.717304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.707684    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:18:15.725335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.721401    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:18:15.725335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:15.721458    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" podUID="04eab7d2-8782-4353-b7b3-1ea5df096979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.020401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.011814    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.021004 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.020965    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.035345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.023359    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.068897 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.068857    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:18:16.069152 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.069123    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.105468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.105421    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:16.202337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.198860    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:16.226899 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.198727    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.227479 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.198779    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 15:18:16.234921 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.198682    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:18:16.237799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.111529    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:18:16.237988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.237957    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.238134 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.222474    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.239081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.239055    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:16.239235 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.239215    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:16.240467 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.240447    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:18:16.267238 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.246823    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.305977 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.270392    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:18:16.318097 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.318054    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.318745 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.318719    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:18:16.323474 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.323436    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.349785 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.349752    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:18:16.778412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.763417    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.890058 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.832365    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:16.890058 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.832755    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 02 15:18:16.890058 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:16.865018    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:17.065360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.048081    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:17.065360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.054600    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:17.065360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.054973    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:17.246571 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.235476    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:17.250086 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.238792    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:17.250539 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.238868    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:17.417904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.238891    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:18:17.418778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.285789    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:17.418980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.418948    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:17.419091 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.401564    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:17.419427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.401519    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:17.419811 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.306499    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:17.419956 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.419933    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:17.531400 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.508831    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:18:17.531792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.531753    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:18:17.531994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.531971    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:18:17.533295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.533270    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:18:17.533516 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.533481    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:17.534538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.534514    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 15:18:17.618844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.618784    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:17.845402 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.820657    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:18:17.894333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.850303    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:17.895038 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.850403    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:18:17.922338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:17.895942    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:18.075773 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.075716    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:18.076171 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.076031    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:18:18.084337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.076060    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 15:18:18.125624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.113401    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:18.160423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.160379    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:18.164408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.153447    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:18.184541 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.178391    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2" event={"ID":"454dc6bb-de62-4247-9127-70bc5d5d0a1e","Type":"ContainerDied","Data":"140bfa1b6d0cc7200afda11a0d2a5a6031bbc4544949ac7feb114bc327ba4cb0"}
Nov 02 15:18:18.209444 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.209401    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="140bfa1b6d0cc7200afda11a0d2a5a6031bbc4544949ac7feb114bc327ba4cb0"
Nov 02 15:18:18.209678 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.179595    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:18.209867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.155677    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:18.215333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.210000    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:18.263490 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.263439    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:18.291341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.267441    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 15:18:18.316336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.274412    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:18.316905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.304506    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-xks-ca\") pod \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\" (UID: \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\") "
Nov 02 15:18:18.332335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.320462    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-openshift-ca\") pod \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\" (UID: \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\") "
Nov 02 15:18:18.347339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.336453    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454dc6bb-de62-4247-9127-70bc5d5d0a1e-kube-api-access-cfc8l\" (UniqueName: \"kubernetes.io/projected/454dc6bb-de62-4247-9127-70bc5d5d0a1e-kube-api-access-cfc8l\") pod \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\" (UID: \"454dc6bb-de62-4247-9127-70bc5d5d0a1e\") "
Nov 02 15:18:18.415337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.311500    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:18.459337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.401553    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-openshift-ca" pod "454dc6bb-de62-4247-9127-70bc5d5d0a1e" (UID: "454dc6bb-de62-4247-9127-70bc5d5d0a1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:18.459337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.451057    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-xks-ca" pod "454dc6bb-de62-4247-9127-70bc5d5d0a1e" (UID: "454dc6bb-de62-4247-9127-70bc5d5d0a1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:18.459337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.448617    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:18:18.459337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.451736    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:18:18.459337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.453759    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 15:18:18.468262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.448560    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:18.505828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.500016    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454dc6bb-de62-4247-9127-70bc5d5d0a1e-kube-api-access-cfc8l" pod "454dc6bb-de62-4247-9127-70bc5d5d0a1e" (UID: "454dc6bb-de62-4247-9127-70bc5d5d0a1e"). InnerVolumeSpecName "kube-api-access-cfc8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:18.505828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.500982    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-openshift-ca\") pod \"307f7aec-bc96-4e78-95db-134ebf285a8e\" (UID: \"307f7aec-bc96-4e78-95db-134ebf285a8e\") "
Nov 02 15:18:18.505828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.501078    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-xks-ca\") pod \"307f7aec-bc96-4e78-95db-134ebf285a8e\" (UID: \"307f7aec-bc96-4e78-95db-134ebf285a8e\") "
Nov 02 15:18:18.505828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.501264    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307f7aec-bc96-4e78-95db-134ebf285a8e-kube-api-access-24z9v\" (UniqueName: \"kubernetes.io/projected/307f7aec-bc96-4e78-95db-134ebf285a8e-kube-api-access-24z9v\") pod \"307f7aec-bc96-4e78-95db-134ebf285a8e\" (UID: \"307f7aec-bc96-4e78-95db-134ebf285a8e\") "
Nov 02 15:18:18.509842 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.509806    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-openshift-ca" pod "307f7aec-bc96-4e78-95db-134ebf285a8e" (UID: "307f7aec-bc96-4e78-95db-134ebf285a8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:18.562595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.562545    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:18.588336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.588233    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:18.588336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.588271    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/454dc6bb-de62-4247-9127-70bc5d5d0a1e-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:18.588336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.588286    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfc8l\" (UniqueName: \"kubernetes.io/projected/454dc6bb-de62-4247-9127-70bc5d5d0a1e-kube-api-access-cfc8l\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:18.588336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.588299    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:18.726597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.623457    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:18.844617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.844577    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307f7aec-bc96-4e78-95db-134ebf285a8e-kube-api-access-24z9v" pod "307f7aec-bc96-4e78-95db-134ebf285a8e" (UID: "307f7aec-bc96-4e78-95db-134ebf285a8e"). InnerVolumeSpecName "kube-api-access-24z9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:18.996391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.916480    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 02 15:18:19.036386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.919265    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:19.036386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.919297    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:18:19.036386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.958363    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:19.036386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:18.978678    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded"
Nov 02 15:18:19.036386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.021359    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24z9v\" (UniqueName: \"kubernetes.io/projected/307f7aec-bc96-4e78-95db-134ebf285a8e-kube-api-access-24z9v\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:19.081399 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.074380    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:18:19.081399 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.074447    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:18:19.081399 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.080262    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:19.117824 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.107327    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:19.118062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.118030    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:19.118230 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.118201    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:19.119063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.119032    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:19.161065 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.161031    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:19.161301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.161272    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:19.236338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.210003    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-xks-ca" pod "307f7aec-bc96-4e78-95db-134ebf285a8e" (UID: "307f7aec-bc96-4e78-95db-134ebf285a8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:19.276390 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.274679    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307f7aec-bc96-4e78-95db-134ebf285a8e-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:19.520344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.495074    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:19.587803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.498540    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:18:19.615335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.549898    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:18:19.615335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.588535    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:18:19.626737 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.626694    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:19.680691 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.650350    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 15:18:19.756338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.736393    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 15:18:19.761888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.748298    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 15:18:19.762394 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.748356    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 15:18:19.805362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:19.804712    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:18:20.020422 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.019936    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:20.033574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.033521    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:20.146108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.064801    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:18:20.146108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.064833    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:20.146108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.089458    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:18:20.146108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.145127    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:20.146108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.145586    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 02 15:18:20.174339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.173245    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:18:20.174339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.173345    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 15:18:20.174762 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.174734    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:20.174973 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.174944    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:20.186091 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.186060    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"
Nov 02 15:18:20.193346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.192770    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28" event={"ID":"307f7aec-bc96-4e78-95db-134ebf285a8e","Type":"ContainerDied","Data":"c80ba7df9ca515aae90c28d2be1183c125274fb265b514ad9ce864022f191b5f"}
Nov 02 15:18:20.193346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.193339    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c80ba7df9ca515aae90c28d2be1183c125274fb265b514ad9ce864022f191b5f"
Nov 02 15:18:20.204798 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.204768    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"
Nov 02 15:18:20.604402 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.604356    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:20.656599 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.656554    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:20.810705 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.810662    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": context deadline exceeded"
Nov 02 15:18:20.872419 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.872380    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:20.872864 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.842484    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 15:18:20.873397 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.842513    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 15:18:20.873711 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.842618    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 15:18:20.880282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.839402    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:18:20.891636 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.890519    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:20.892196 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.892171    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:18:20.894867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.892441    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:18:20.894867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.892517    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:18:20.894867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.892554    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:18:20.894867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.892681    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:18:20.894867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.892719    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:18:20.894867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.892850    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:18:20.903869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.903847    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:18:20.906555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.905489    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:18:20.916853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.915363    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:18:20.928330 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.924146    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:18:20.933329 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.931719    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:18:20.947483 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.943095    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:18:20.971344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:20.969808    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:18:21.108194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:21.107988    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 15:18:21.108194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:21.108043    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 15:18:21.160242 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:21.160201    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:21.960577 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:21.960540    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"]
Nov 02 15:18:21.961761 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:21.961718    4891 cpu_manager.go:414] "containerMap: removing container" podUID="454dc6bb-de62-4247-9127-70bc5d5d0a1e" containerName="revoke-keys"
Nov 02 15:18:21.961919 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:21.961905    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454dc6bb-de62-4247-9127-70bc5d5d0a1e" containerName="revoke-keys"
Nov 02 15:18:21.962075 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:21.962064    4891 cpu_manager.go:414] "containerMap: removing container" podUID="307f7aec-bc96-4e78-95db-134ebf285a8e" containerName="revoke-keys"
Nov 02 15:18:21.962154 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:21.962145    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307f7aec-bc96-4e78-95db-134ebf285a8e" containerName="revoke-keys"
Nov 02 15:18:21.962452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:21.962437    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="454dc6bb-de62-4247-9127-70bc5d5d0a1e" containerName="revoke-keys"
Nov 02 15:18:21.962574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:21.962562    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="307f7aec-bc96-4e78-95db-134ebf285a8e" containerName="revoke-keys"
Nov 02 15:18:21.975740 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:21.975710    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.029832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.029790    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"]
Nov 02 15:18:22.102250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.102188    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.102545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.102519    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.102699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.102682    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.102860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.102844    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40993be1-454f-4d83-8b5a-85c1975f71da-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.102976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.102948    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40993be1-454f-4d83-8b5a-85c1975f71da-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.103221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.103157    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.103499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.103480    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6k9c\" (UniqueName: \"kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-kube-api-access-h6k9c\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.103643 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.103626    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.103855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.103839    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.124390 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.108384    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:22.207248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.207190    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.207622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.207602    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.207759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.207745    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.207916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.207886    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.208090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.208075    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40993be1-454f-4d83-8b5a-85c1975f71da-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.208184 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.208172    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40993be1-454f-4d83-8b5a-85c1975f71da-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.208287 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.208273    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.208557 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.208541    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6k9c\" (UniqueName: \"kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-kube-api-access-h6k9c\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.208663 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.208650    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.209202 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.209181    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-certs\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.210418 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.210395    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-credential-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.227867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.211044    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-data\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.227867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.224404    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:18:22.227867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.227355    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-socket\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.227867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.227356    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:22.227867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.226814    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-envoy\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.227867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.227045    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40993be1-454f-4d83-8b5a-85c1975f71da-istio-podinfo\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.227867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.227602    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40993be1-454f-4d83-8b5a-85c1975f71da-istiod-ca-cert\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.293914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.293877    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-istio-token\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.337061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.336879    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6k9c\" (UniqueName: \"kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-kube-api-access-h6k9c\") pod \"e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") " pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:22.610937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:22.610860    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:23.027530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.027488    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:23.028839 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.028814    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:18:23.120329 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.119347    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"]
Nov 02 15:18:23.139882 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.139837    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:23.140295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.139928    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:23.140295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.139974    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:23.146504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.146465    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:18:23.146617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.146515    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:23.270334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.267434    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh" event={"ID":"40993be1-454f-4d83-8b5a-85c1975f71da","Type":"ContainerStarted","Data":"ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a"}
Nov 02 15:18:23.270334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.267495    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh" event={"ID":"40993be1-454f-4d83-8b5a-85c1975f71da","Type":"ContainerStarted","Data":"ee31be4ee17016bb1d11a9cf93ba18f02748ea834a219b643863767cd02175d9"}
Nov 02 15:18:23.413388 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.409175    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh" podStartSLOduration=2.409154455 podStartE2EDuration="2.409154455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:23.408496134 +0000 UTC m=+1311.001112944" watchObservedRunningTime="2026-11-02 15:18:23.409154455 +0000 UTC m=+1311.001771255"
Nov 02 15:18:23.612107 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.612014    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:23.873677 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.873640    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:23.883332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:23.876679    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:18:24.612385 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:24.612348    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 15:18:24.612776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:24.612403    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh" podUID="40993be1-454f-4d83-8b5a-85c1975f71da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 15:18:25.611460 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.611426    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 15:18:25.611643 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.611479    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh" podUID="40993be1-454f-4d83-8b5a-85c1975f71da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 15:18:25.689913 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.689873    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"]
Nov 02 15:18:25.693978 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.693958    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:25.712059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.712028    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1414d6c4-dbwb4\" (UID: \"b4ae8c6f-6385-4221-9f82-094ca468bec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:25.712206 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.712181    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5m5\" (UniqueName: \"kubernetes.io/projected/b4ae8c6f-6385-4221-9f82-094ca468bec0-kube-api-access-rl5m5\") pod \"maas-api-revoke-sub-1414d6c4-dbwb4\" (UID: \"b4ae8c6f-6385-4221-9f82-094ca468bec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:25.712282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.712235    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1414d6c4-dbwb4\" (UID: \"b4ae8c6f-6385-4221-9f82-094ca468bec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:25.756455 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.756418    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:18:25.756646 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.756539    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:18:25.756646 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.756614    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:18:25.756814 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.756792    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:18:25.780336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.778863    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"]
Nov 02 15:18:25.813959 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.813923    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1414d6c4-dbwb4\" (UID: \"b4ae8c6f-6385-4221-9f82-094ca468bec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:25.814177 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.814157    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1414d6c4-dbwb4\" (UID: \"b4ae8c6f-6385-4221-9f82-094ca468bec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:25.814355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.814336    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5m5\" (UniqueName: \"kubernetes.io/projected/b4ae8c6f-6385-4221-9f82-094ca468bec0-kube-api-access-rl5m5\") pod \"maas-api-revoke-sub-1414d6c4-dbwb4\" (UID: \"b4ae8c6f-6385-4221-9f82-094ca468bec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:25.815478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.815453    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1414d6c4-dbwb4\" (UID: \"b4ae8c6f-6385-4221-9f82-094ca468bec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:25.818686 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.818167    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1414d6c4-dbwb4\" (UID: \"b4ae8c6f-6385-4221-9f82-094ca468bec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:25.897105 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:25.896995    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5m5\" (UniqueName: \"kubernetes.io/projected/b4ae8c6f-6385-4221-9f82-094ca468bec0-kube-api-access-rl5m5\") pod \"maas-api-revoke-sub-1414d6c4-dbwb4\" (UID: \"b4ae8c6f-6385-4221-9f82-094ca468bec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:26.012345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:26.012201    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:26.582796 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:26.582759    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"]
Nov 02 15:18:26.611825 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:26.611798    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 15:18:26.611941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:26.611846    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh" podUID="40993be1-454f-4d83-8b5a-85c1975f71da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 15:18:27.333924 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:27.333830    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4" event={"ID":"b4ae8c6f-6385-4221-9f82-094ca468bec0","Type":"ContainerStarted","Data":"55215f96bda58e4c52bba1c647274a4df5ac7d5eb4a86478456f025f76af32f4"}
Nov 02 15:18:27.333924 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:27.333886    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4" event={"ID":"b4ae8c6f-6385-4221-9f82-094ca468bec0","Type":"ContainerStarted","Data":"d7f3250108de2b272735009749d9026795e0401d55bf967d5c3fe487add60917"}
Nov 02 15:18:27.475192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:27.475125    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4" podStartSLOduration=2.475105525 podStartE2EDuration="2.475105525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:27.472213133 +0000 UTC m=+1315.064829933" watchObservedRunningTime="2026-11-02 15:18:27.475105525 +0000 UTC m=+1315.067722325"
Nov 02 15:18:27.611687 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:27.611599    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 15:18:27.611687 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:27.611666    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh" podUID="40993be1-454f-4d83-8b5a-85c1975f71da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 15:18:28.374356 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:28.351268    4891 generic.go:356] "Generic (PLEG): container finished" podID="b4ae8c6f-6385-4221-9f82-094ca468bec0" containerID="55215f96bda58e4c52bba1c647274a4df5ac7d5eb4a86478456f025f76af32f4" exitCode=0
Nov 02 15:18:28.374356 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:28.361783    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4" event={"ID":"b4ae8c6f-6385-4221-9f82-094ca468bec0","Type":"ContainerDied","Data":"55215f96bda58e4c52bba1c647274a4df5ac7d5eb4a86478456f025f76af32f4"}
Nov 02 15:18:28.611844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:28.611811    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 15:18:28.612025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:28.611867    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh" podUID="40993be1-454f-4d83-8b5a-85c1975f71da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 15:18:29.563277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:29.563250    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:29.611285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:29.611249    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 15:18:29.611494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:29.611324    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh" podUID="40993be1-454f-4d83-8b5a-85c1975f71da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 15:18:29.625480 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:29.625449    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-openshift-ca\") pod \"b4ae8c6f-6385-4221-9f82-094ca468bec0\" (UID: \"b4ae8c6f-6385-4221-9f82-094ca468bec0\") "
Nov 02 15:18:29.625673 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:29.625500    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-xks-ca\") pod \"b4ae8c6f-6385-4221-9f82-094ca468bec0\" (UID: \"b4ae8c6f-6385-4221-9f82-094ca468bec0\") "
Nov 02 15:18:29.625673 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:29.625593    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ae8c6f-6385-4221-9f82-094ca468bec0-kube-api-access-rl5m5\" (UniqueName: \"kubernetes.io/projected/b4ae8c6f-6385-4221-9f82-094ca468bec0-kube-api-access-rl5m5\") pod \"b4ae8c6f-6385-4221-9f82-094ca468bec0\" (UID: \"b4ae8c6f-6385-4221-9f82-094ca468bec0\") "
Nov 02 15:18:29.626748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:29.626716    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-openshift-ca" pod "b4ae8c6f-6385-4221-9f82-094ca468bec0" (UID: "b4ae8c6f-6385-4221-9f82-094ca468bec0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:29.632497 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:29.632428    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-xks-ca" pod "b4ae8c6f-6385-4221-9f82-094ca468bec0" (UID: "b4ae8c6f-6385-4221-9f82-094ca468bec0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:29.632497 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:29.632481    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ae8c6f-6385-4221-9f82-094ca468bec0-kube-api-access-rl5m5" pod "b4ae8c6f-6385-4221-9f82-094ca468bec0" (UID: "b4ae8c6f-6385-4221-9f82-094ca468bec0"). InnerVolumeSpecName "kube-api-access-rl5m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:29.728454 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:29.728416    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:29.728641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:29.728481    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl5m5\" (UniqueName: \"kubernetes.io/projected/b4ae8c6f-6385-4221-9f82-094ca468bec0-kube-api-access-rl5m5\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:29.728641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:29.728497    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ae8c6f-6385-4221-9f82-094ca468bec0-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:30.391554 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:30.391518    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"
Nov 02 15:18:30.391779 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:30.391548    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4" event={"ID":"b4ae8c6f-6385-4221-9f82-094ca468bec0","Type":"ContainerDied","Data":"d7f3250108de2b272735009749d9026795e0401d55bf967d5c3fe487add60917"}
Nov 02 15:18:30.391779 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:30.391630    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7f3250108de2b272735009749d9026795e0401d55bf967d5c3fe487add60917"
Nov 02 15:18:31.086696 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:31.086660    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:31.087246 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:31.087165    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:31.088063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:31.088042    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:18:32.564527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.564488    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"]
Nov 02 15:18:32.565295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.565274    4891 cpu_manager.go:414] "containerMap: removing container" podUID="b4ae8c6f-6385-4221-9f82-094ca468bec0" containerName="revoke-keys"
Nov 02 15:18:32.565450 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.565298    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ae8c6f-6385-4221-9f82-094ca468bec0" containerName="revoke-keys"
Nov 02 15:18:32.565527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.565511    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ae8c6f-6385-4221-9f82-094ca468bec0" containerName="revoke-keys"
Nov 02 15:18:32.569458 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.569432    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:32.621869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.621828    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:18:32.627407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.627380    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"]
Nov 02 15:18:32.669566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.669539    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:18:32.671187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.671165    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:18:32.671187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.671179    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:18:32.671407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.671213    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87dxg\" (UniqueName: \"kubernetes.io/projected/b91c5ffc-c65a-4b39-bae9-09769108d54e-kube-api-access-87dxg\") pod \"maas-api-revoke-sub-b3cb7a2c-mcbww\" (UID: \"b91c5ffc-c65a-4b39-bae9-09769108d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:32.671549 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.671530    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3cb7a2c-mcbww\" (UID: \"b91c5ffc-c65a-4b39-bae9-09769108d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:32.671612 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.671570    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3cb7a2c-mcbww\" (UID: \"b91c5ffc-c65a-4b39-bae9-09769108d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:32.772599 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.772568    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3cb7a2c-mcbww\" (UID: \"b91c5ffc-c65a-4b39-bae9-09769108d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:32.772767 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.772605    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3cb7a2c-mcbww\" (UID: \"b91c5ffc-c65a-4b39-bae9-09769108d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:32.772767 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.772680    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87dxg\" (UniqueName: \"kubernetes.io/projected/b91c5ffc-c65a-4b39-bae9-09769108d54e-kube-api-access-87dxg\") pod \"maas-api-revoke-sub-b3cb7a2c-mcbww\" (UID: \"b91c5ffc-c65a-4b39-bae9-09769108d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:32.773459 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.773433    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3cb7a2c-mcbww\" (UID: \"b91c5ffc-c65a-4b39-bae9-09769108d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:32.774988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.774958    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3cb7a2c-mcbww\" (UID: \"b91c5ffc-c65a-4b39-bae9-09769108d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:32.833683 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.833610    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87dxg\" (UniqueName: \"kubernetes.io/projected/b91c5ffc-c65a-4b39-bae9-09769108d54e-kube-api-access-87dxg\") pod \"maas-api-revoke-sub-b3cb7a2c-mcbww\" (UID: \"b91c5ffc-c65a-4b39-bae9-09769108d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:32.855031 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.854996    4891 scope.go:122] "RemoveContainer" containerID="74378803719b4c251f23157cb4260b28c3593894fe155a711115ebbdbaa54414"
Nov 02 15:18:32.866393 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.866371    4891 scope.go:122] "RemoveContainer" containerID="6bfdef5f85e2b598549e50b96c6ec35b357fbde3992719431dd18d25ca06179a"
Nov 02 15:18:32.876509 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.876491    4891 scope.go:122] "RemoveContainer" containerID="75d7ce1653667d0bf5a8ed974473bbfca2247655e503af2381147a19d13fb929"
Nov 02 15:18:32.887475 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.887449    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:32.889484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.889462    4891 scope.go:122] "RemoveContainer" containerID="1bd32e374cfe9d80138cf38887f0b4584af26291cbeee8ab04d4e457d6ca7bdf"
Nov 02 15:18:32.909108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.909072    4891 scope.go:122] "RemoveContainer" containerID="1ec33e1af921a898417f7526f0c455ac3089be18edb5637645e1aac117a48475"
Nov 02 15:18:32.938991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.938963    4891 scope.go:122] "RemoveContainer" containerID="e24cf6c3bb2c3f13710fa0805cc81c59eddccd82fc5f01780442443256d62c6c"
Nov 02 15:18:32.953017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.952996    4891 scope.go:122] "RemoveContainer" containerID="8d0f7d576cd6cb72f0a43c1bd32eb266bd11cbc3d3f1c7aea68855bed1ce0b26"
Nov 02 15:18:32.971716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.971668    4891 scope.go:122] "RemoveContainer" containerID="91d55f249045a69654dfa62dd1035dd0b6b65807b432922c3c27a7eb397f4828"
Nov 02 15:18:32.985597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.985572    4891 scope.go:122] "RemoveContainer" containerID="53adb112e663dbaca3e1a2ea8fad9c6622261690f9bc2d3f407aac8d0faddaae"
Nov 02 15:18:32.998855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:32.998810    4891 scope.go:122] "RemoveContainer" containerID="ec69b803ebe193f27a5eedb9d1b3fd7fd85a5ff42df1e135fd4951a1c6ec0779"
Nov 02 15:18:33.009612 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:33.009590    4891 scope.go:122] "RemoveContainer" containerID="255b8a5d93c0dfad97056f15c3037db95c88f0a79efb9fec72b0a759afe3e7d3"
Nov 02 15:18:33.081890 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:33.081861    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"]
Nov 02 15:18:33.095595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:33.095566    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d38bc66-4lfbr"]
Nov 02 15:18:33.293041 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:18:33.293008    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91c5ffc_c65a_4b39_bae9_09769108d54e.slice/crio-00dedaf7ba720d0f4918492513d5a877244cc30cb9e1931517830f4c17941ae1 WatchSource:0}: Error finding container 00dedaf7ba720d0f4918492513d5a877244cc30cb9e1931517830f4c17941ae1: Status 404 returned error can't find the container with id 00dedaf7ba720d0f4918492513d5a877244cc30cb9e1931517830f4c17941ae1
Nov 02 15:18:33.293626 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:33.293604    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"]
Nov 02 15:18:33.430652 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:33.430551    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww" event={"ID":"b91c5ffc-c65a-4b39-bae9-09769108d54e","Type":"ContainerStarted","Data":"00dedaf7ba720d0f4918492513d5a877244cc30cb9e1931517830f4c17941ae1"}
Nov 02 15:18:34.441612 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:34.441575    4891 generic.go:356] "Generic (PLEG): container finished" podID="b91c5ffc-c65a-4b39-bae9-09769108d54e" containerID="c91c5e3441a82c726b1921dd23af53fa28a6850d7296fad4e6e0766d98430d9f" exitCode=0
Nov 02 15:18:34.442088 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:34.441670    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww" event={"ID":"b91c5ffc-c65a-4b39-bae9-09769108d54e","Type":"ContainerDied","Data":"c91c5e3441a82c726b1921dd23af53fa28a6850d7296fad4e6e0766d98430d9f"}
Nov 02 15:18:34.523384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:34.523354    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e1a797-bf38-4019-a144-1ee24dd8ce60" path="/var/lib/kubelet/pods/96e1a797-bf38-4019-a144-1ee24dd8ce60/volumes"
Nov 02 15:18:35.631590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:35.631566    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:35.716868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:35.716785    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-openshift-ca\") pod \"b91c5ffc-c65a-4b39-bae9-09769108d54e\" (UID: \"b91c5ffc-c65a-4b39-bae9-09769108d54e\") "
Nov 02 15:18:35.716868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:35.716856    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-xks-ca\") pod \"b91c5ffc-c65a-4b39-bae9-09769108d54e\" (UID: \"b91c5ffc-c65a-4b39-bae9-09769108d54e\") "
Nov 02 15:18:35.717047 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:35.716941    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91c5ffc-c65a-4b39-bae9-09769108d54e-kube-api-access-87dxg\" (UniqueName: \"kubernetes.io/projected/b91c5ffc-c65a-4b39-bae9-09769108d54e-kube-api-access-87dxg\") pod \"b91c5ffc-c65a-4b39-bae9-09769108d54e\" (UID: \"b91c5ffc-c65a-4b39-bae9-09769108d54e\") "
Nov 02 15:18:35.717239 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:35.717213    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-openshift-ca" pod "b91c5ffc-c65a-4b39-bae9-09769108d54e" (UID: "b91c5ffc-c65a-4b39-bae9-09769108d54e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:35.717460 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:35.717436    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:35.719090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:35.719062    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-xks-ca" pod "b91c5ffc-c65a-4b39-bae9-09769108d54e" (UID: "b91c5ffc-c65a-4b39-bae9-09769108d54e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:35.719090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:35.719070    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91c5ffc-c65a-4b39-bae9-09769108d54e-kube-api-access-87dxg" pod "b91c5ffc-c65a-4b39-bae9-09769108d54e" (UID: "b91c5ffc-c65a-4b39-bae9-09769108d54e"). InnerVolumeSpecName "kube-api-access-87dxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:35.819359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:35.819296    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91c5ffc-c65a-4b39-bae9-09769108d54e-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:35.819359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:35.819354    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87dxg\" (UniqueName: \"kubernetes.io/projected/b91c5ffc-c65a-4b39-bae9-09769108d54e-kube-api-access-87dxg\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:36.461949 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.461914    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"
Nov 02 15:18:36.462128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.461947    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww" event={"ID":"b91c5ffc-c65a-4b39-bae9-09769108d54e","Type":"ContainerDied","Data":"00dedaf7ba720d0f4918492513d5a877244cc30cb9e1931517830f4c17941ae1"}
Nov 02 15:18:36.462128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.461993    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00dedaf7ba720d0f4918492513d5a877244cc30cb9e1931517830f4c17941ae1"
Nov 02 15:18:36.732851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.732759    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"]
Nov 02 15:18:36.733637 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.733614    4891 cpu_manager.go:414] "containerMap: removing container" podUID="b91c5ffc-c65a-4b39-bae9-09769108d54e" containerName="revoke-keys"
Nov 02 15:18:36.733637 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.733637    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91c5ffc-c65a-4b39-bae9-09769108d54e" containerName="revoke-keys"
Nov 02 15:18:36.733809 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.733789    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91c5ffc-c65a-4b39-bae9-09769108d54e" containerName="revoke-keys"
Nov 02 15:18:36.737672 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.737649    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.752187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.752163    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"]
Nov 02 15:18:36.756342 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.756302    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.832198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.832143    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"]
Nov 02 15:18:36.857423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.857362    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"]
Nov 02 15:18:36.859283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.859241    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.859467 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.859335    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.859467 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.859410    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.859918 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.859889    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1559e94a-36ed-40a4-8118-3d8f0f338a17-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.860274 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860224    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brc8v\" (UniqueName: \"kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-kube-api-access-brc8v\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.860422 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860298    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.860422 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860357    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw8dx\" (UniqueName: \"kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-kube-api-access-tw8dx\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.860422 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860403    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.860607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860468    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.860607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860577    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.860714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860637    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.860714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860676    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.860714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860707    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.860891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860737    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.860891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860771    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.860891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860814    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.860891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860868    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.861107 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.860925    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.961777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.961738    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.961777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.961784    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1559e94a-36ed-40a4-8118-3d8f0f338a17-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.961973 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.961808    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brc8v\" (UniqueName: \"kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-kube-api-access-brc8v\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.961973 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.961845    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.961973 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.961876    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw8dx\" (UniqueName: \"kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-kube-api-access-tw8dx\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.961973 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.961916    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.962199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.961972    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.962199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962071    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.962199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962122    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.962199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962157    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.962199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962184    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.962437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962209    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.962437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962238    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.962437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962277    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.962437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962349    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.962437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962401    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.962437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962425    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.962672 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962478    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.962672 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962527    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1559e94a-36ed-40a4-8118-3d8f0f338a17-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.962672 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962570    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.962672 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962632    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.962887 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.962860    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.963189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.963160    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.963291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.963210    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.963492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.963470    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.963586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.963563    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.964292 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.963714    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.964292 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.963781    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.965354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.965302    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.965550 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.965528    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:36.965825 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.965798    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:36.965965 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:36.965949    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:37.025545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.025488    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:37.025743 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.025574    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw8dx\" (UniqueName: \"kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-kube-api-access-tw8dx\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:37.025743 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.025665    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brc8v\" (UniqueName: \"kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-kube-api-access-brc8v\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:37.026727 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.026708    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:37.052130 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.052106    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:37.070617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.070591    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:37.547894 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:18:37.514142    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc3c1fb_c245_43ca_b5f4_4ea9c08b3768.slice/crio-e51d08164547190bdec520291443d7e2689a6e2a013dcbddff1315d9fafcda56 WatchSource:0}: Error finding container e51d08164547190bdec520291443d7e2689a6e2a013dcbddff1315d9fafcda56: Status 404 returned error can't find the container with id e51d08164547190bdec520291443d7e2689a6e2a013dcbddff1315d9fafcda56
Nov 02 15:18:37.547894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.515965    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"]
Nov 02 15:18:37.547894 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:18:37.517187    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1559e94a_36ed_40a4_8118_3d8f0f338a17.slice/crio-18e48912dcc6a04e997f5927ac7cfa7301ba67f956bfac1d576247e315e4f657 WatchSource:0}: Error finding container 18e48912dcc6a04e997f5927ac7cfa7301ba67f956bfac1d576247e315e4f657: Status 404 returned error can't find the container with id 18e48912dcc6a04e997f5927ac7cfa7301ba67f956bfac1d576247e315e4f657
Nov 02 15:18:37.547894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.517216    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:37.547894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.517281    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:37.547894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.517327    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:37.547894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.519525    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:37.547894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.519574    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:37.547894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.519602    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:37.547894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.522677    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"]
Nov 02 15:18:37.883224 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.883188    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"]
Nov 02 15:18:37.888925 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.888897    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:37.959363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.955596    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:18:37.959363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.955596    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:18:37.959363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.955648    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:18:37.959363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.955741    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:18:37.959363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:37.959157    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"]
Nov 02 15:18:38.112838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.107721    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pf2c\" (UniqueName: \"kubernetes.io/projected/d38b34cf-4981-4932-82f8-b3d2856a5c00-kube-api-access-9pf2c\") pod \"maas-api-revoke-sub-e77aaa20-6v8gv\" (UID: \"d38b34cf-4981-4932-82f8-b3d2856a5c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:38.112838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.107792    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e77aaa20-6v8gv\" (UID: \"d38b34cf-4981-4932-82f8-b3d2856a5c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:38.112838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.107980    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e77aaa20-6v8gv\" (UID: \"d38b34cf-4981-4932-82f8-b3d2856a5c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:38.215408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.212300    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pf2c\" (UniqueName: \"kubernetes.io/projected/d38b34cf-4981-4932-82f8-b3d2856a5c00-kube-api-access-9pf2c\") pod \"maas-api-revoke-sub-e77aaa20-6v8gv\" (UID: \"d38b34cf-4981-4932-82f8-b3d2856a5c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:38.215408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.214482    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e77aaa20-6v8gv\" (UID: \"d38b34cf-4981-4932-82f8-b3d2856a5c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:38.215703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.215509    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e77aaa20-6v8gv\" (UID: \"d38b34cf-4981-4932-82f8-b3d2856a5c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:38.215703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.215628    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e77aaa20-6v8gv\" (UID: \"d38b34cf-4981-4932-82f8-b3d2856a5c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:38.242331 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.228089    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e77aaa20-6v8gv\" (UID: \"d38b34cf-4981-4932-82f8-b3d2856a5c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:38.288392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.287340    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pf2c\" (UniqueName: \"kubernetes.io/projected/d38b34cf-4981-4932-82f8-b3d2856a5c00-kube-api-access-9pf2c\") pod \"maas-api-revoke-sub-e77aaa20-6v8gv\" (UID: \"d38b34cf-4981-4932-82f8-b3d2856a5c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:38.518390 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.515739    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:38.639013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.638275    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7" event={"ID":"1559e94a-36ed-40a4-8118-3d8f0f338a17","Type":"ContainerStarted","Data":"77c95b8bc043509e7aebb8232ffadb7e8f2709821949f66c497af131683c7480"}
Nov 02 15:18:38.639013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.638363    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7" event={"ID":"1559e94a-36ed-40a4-8118-3d8f0f338a17","Type":"ContainerStarted","Data":"18e48912dcc6a04e997f5927ac7cfa7301ba67f956bfac1d576247e315e4f657"}
Nov 02 15:18:38.665340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.646892    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt" event={"ID":"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768","Type":"ContainerStarted","Data":"db8f9da6aa43f6c558b03328af65bc14936442e841d87c0aebe3159bf432d7a1"}
Nov 02 15:18:38.665340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.649165    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt" event={"ID":"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768","Type":"ContainerStarted","Data":"e51d08164547190bdec520291443d7e2689a6e2a013dcbddff1315d9fafcda56"}
Nov 02 15:18:38.815282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:38.815211    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7" podStartSLOduration=2.815187088 podStartE2EDuration="2.815187088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:38.773708282 +0000 UTC m=+1326.366325121" watchObservedRunningTime="2026-11-02 15:18:38.815187088 +0000 UTC m=+1326.407803902"
Nov 02 15:18:39.138870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:39.138771    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:39.139407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:39.139040    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:39.202367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:39.166297    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 15:18:39.202367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:39.166373    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt" podUID="dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 15:18:39.202367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:39.166687    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 15:18:39.202367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:39.166728    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7" podUID="1559e94a-36ed-40a4-8118-3d8f0f338a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 15:18:39.225347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:39.215368    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt" podStartSLOduration=3.215347023 podStartE2EDuration="3.215347023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:38.924398051 +0000 UTC m=+1326.517014851" watchObservedRunningTime="2026-11-02 15:18:39.215347023 +0000 UTC m=+1326.807963823"
Nov 02 15:18:39.632550 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:39.632457    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"]
Nov 02 15:18:39.733967 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:39.733920    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv" event={"ID":"d38b34cf-4981-4932-82f8-b3d2856a5c00","Type":"ContainerStarted","Data":"eaf1a62658851bce948ac4fb5dc4d28fb504e80dce02a47b1e245be173c8add3"}
Nov 02 15:18:40.083336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:40.067473    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 15:18:40.083336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:40.067536    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt" podUID="dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 15:18:40.083336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:40.071587    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 15:18:40.083336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:40.071628    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7" podUID="1559e94a-36ed-40a4-8118-3d8f0f338a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 15:18:40.764725 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:40.764363    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv" event={"ID":"d38b34cf-4981-4932-82f8-b3d2856a5c00","Type":"ContainerStarted","Data":"65f4fe656a15ba8cdaa8b2a921c828434e26a3d9befc7b32eb9b976406fc19b0"}
Nov 02 15:18:40.909332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:40.908769    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv" podStartSLOduration=3.908750436 podStartE2EDuration="3.908750436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:40.899379274 +0000 UTC m=+1328.491996081" watchObservedRunningTime="2026-11-02 15:18:40.908750436 +0000 UTC m=+1328.501367240"
Nov 02 15:18:41.071339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:41.059918    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 15:18:41.071339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:41.059981    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt" podUID="dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 15:18:41.090338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:41.077144    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 15:18:41.090338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:41.077219    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7" podUID="1559e94a-36ed-40a4-8118-3d8f0f338a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 15:18:41.130696 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:18:41.130213    4891 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38b34cf_4981_4932_82f8_b3d2856a5c00.slice/crio-65f4fe656a15ba8cdaa8b2a921c828434e26a3d9befc7b32eb9b976406fc19b0.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:18:41.833752 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:41.833700    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"]
Nov 02 15:18:41.840338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:41.840272    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"
Nov 02 15:18:41.840512 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:41.840392    4891 generic.go:356] "Generic (PLEG): container finished" podID="d38b34cf-4981-4932-82f8-b3d2856a5c00" containerID="65f4fe656a15ba8cdaa8b2a921c828434e26a3d9befc7b32eb9b976406fc19b0" exitCode=0
Nov 02 15:18:41.840683 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:41.840664    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv" event={"ID":"d38b34cf-4981-4932-82f8-b3d2856a5c00","Type":"ContainerDied","Data":"65f4fe656a15ba8cdaa8b2a921c828434e26a3d9befc7b32eb9b976406fc19b0"}
Nov 02 15:18:41.937372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:41.936232    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"]
Nov 02 15:18:41.965335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:41.964552    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5ccdeb27\""
Nov 02 15:18:41.965335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:41.964632    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5ccdeb27-dockercfg-4hs8h\""
Nov 02 15:18:42.037332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.036572    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebebeb8-f4ed-47d6-a491-ceddc58fc949-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv\" (UID: \"bebebeb8-f4ed-47d6-a491-ceddc58fc949\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"
Nov 02 15:18:42.037332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.036681    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzplp\" (UniqueName: \"kubernetes.io/projected/bebebeb8-f4ed-47d6-a491-ceddc58fc949-kube-api-access-wzplp\") pod \"payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv\" (UID: \"bebebeb8-f4ed-47d6-a491-ceddc58fc949\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"
Nov 02 15:18:42.129431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.121468    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 15:18:42.129431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.121533    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7" podUID="1559e94a-36ed-40a4-8118-3d8f0f338a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 15:18:42.129431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.121987    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 15:18:42.129431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.122037    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt" podUID="dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 15:18:42.142127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.141277    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebebeb8-f4ed-47d6-a491-ceddc58fc949-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv\" (UID: \"bebebeb8-f4ed-47d6-a491-ceddc58fc949\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"
Nov 02 15:18:42.142127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.141403    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzplp\" (UniqueName: \"kubernetes.io/projected/bebebeb8-f4ed-47d6-a491-ceddc58fc949-kube-api-access-wzplp\") pod \"payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv\" (UID: \"bebebeb8-f4ed-47d6-a491-ceddc58fc949\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"
Nov 02 15:18:42.225155 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.172428    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebebeb8-f4ed-47d6-a491-ceddc58fc949-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv\" (UID: \"bebebeb8-f4ed-47d6-a491-ceddc58fc949\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"
Nov 02 15:18:42.232855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.232816    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"]
Nov 02 15:18:42.343845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.343809    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:42.372333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.360162    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzplp\" (UniqueName: \"kubernetes.io/projected/bebebeb8-f4ed-47d6-a491-ceddc58fc949-kube-api-access-wzplp\") pod \"payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv\" (UID: \"bebebeb8-f4ed-47d6-a491-ceddc58fc949\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"
Nov 02 15:18:42.372333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.362848    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20c4aa69-sgcfq\" (UID: \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:42.372333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.363088    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6mzw\" (UniqueName: \"kubernetes.io/projected/a555e6ba-dc5d-45ac-89da-ed00a0205f65-kube-api-access-j6mzw\") pod \"maas-api-revoke-sub-20c4aa69-sgcfq\" (UID: \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:42.372333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.363232    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20c4aa69-sgcfq\" (UID: \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:42.373751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.373719    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"]
Nov 02 15:18:42.491338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.464346    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20c4aa69-sgcfq\" (UID: \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:42.526408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.525648    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6mzw\" (UniqueName: \"kubernetes.io/projected/a555e6ba-dc5d-45ac-89da-ed00a0205f65-kube-api-access-j6mzw\") pod \"maas-api-revoke-sub-20c4aa69-sgcfq\" (UID: \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:42.526408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.525786    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20c4aa69-sgcfq\" (UID: \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:42.526657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.492440    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20c4aa69-sgcfq\" (UID: \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:42.529689 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.529663    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20c4aa69-sgcfq\" (UID: \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:42.542591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.542560    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"
Nov 02 15:18:42.617215 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.617172    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6mzw\" (UniqueName: \"kubernetes.io/projected/a555e6ba-dc5d-45ac-89da-ed00a0205f65-kube-api-access-j6mzw\") pod \"maas-api-revoke-sub-20c4aa69-sgcfq\" (UID: \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:42.712116 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:42.712028    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:43.077942 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.068731    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 15:18:43.077942 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.068799    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt" podUID="dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 15:18:43.110237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.110186    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 15:18:43.110445 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.110256    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7" podUID="1559e94a-36ed-40a4-8118-3d8f0f338a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 15:18:43.209771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.209739    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"]
Nov 02 15:18:43.300385 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.300353    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"]
Nov 02 15:18:43.301582 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:18:43.301553    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebebeb8_f4ed_47d6_a491_ceddc58fc949.slice/crio-d1a5bd5af261da2474c45f7916eac85ba0a4b55ea0a72e8668596520a196c93f WatchSource:0}: Error finding container d1a5bd5af261da2474c45f7916eac85ba0a4b55ea0a72e8668596520a196c93f: Status 404 returned error can't find the container with id d1a5bd5af261da2474c45f7916eac85ba0a4b55ea0a72e8668596520a196c93f
Nov 02 15:18:43.403928 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.403850    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"]
Nov 02 15:18:43.409991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.409783    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:43.415822 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.415800    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:43.427429 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.427399    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-xks-ca\") pod \"d38b34cf-4981-4932-82f8-b3d2856a5c00\" (UID: \"d38b34cf-4981-4932-82f8-b3d2856a5c00\") "
Nov 02 15:18:43.427560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.427465    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38b34cf-4981-4932-82f8-b3d2856a5c00-kube-api-access-9pf2c\" (UniqueName: \"kubernetes.io/projected/d38b34cf-4981-4932-82f8-b3d2856a5c00-kube-api-access-9pf2c\") pod \"d38b34cf-4981-4932-82f8-b3d2856a5c00\" (UID: \"d38b34cf-4981-4932-82f8-b3d2856a5c00\") "
Nov 02 15:18:43.427560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.427496    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-openshift-ca\") pod \"d38b34cf-4981-4932-82f8-b3d2856a5c00\" (UID: \"d38b34cf-4981-4932-82f8-b3d2856a5c00\") "
Nov 02 15:18:43.427816 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.427794    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b6bf862-nnpqd\" (UID: \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:43.427876 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.427846    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5s45\" (UniqueName: \"kubernetes.io/projected/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-kube-api-access-p5s45\") pod \"maas-api-revoke-sub-9b6bf862-nnpqd\" (UID: \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:43.428180 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.428156    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b6bf862-nnpqd\" (UID: \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:43.428255 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.428163    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-openshift-ca" pod "d38b34cf-4981-4932-82f8-b3d2856a5c00" (UID: "d38b34cf-4981-4932-82f8-b3d2856a5c00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:43.428336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.428305    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:43.467335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.459458    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38b34cf-4981-4932-82f8-b3d2856a5c00-kube-api-access-9pf2c" pod "d38b34cf-4981-4932-82f8-b3d2856a5c00" (UID: "d38b34cf-4981-4932-82f8-b3d2856a5c00"). InnerVolumeSpecName "kube-api-access-9pf2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:43.467335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.459560    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-xks-ca" pod "d38b34cf-4981-4932-82f8-b3d2856a5c00" (UID: "d38b34cf-4981-4932-82f8-b3d2856a5c00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:43.501841 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.501810    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"]
Nov 02 15:18:43.529148 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.529117    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5s45\" (UniqueName: \"kubernetes.io/projected/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-kube-api-access-p5s45\") pod \"maas-api-revoke-sub-9b6bf862-nnpqd\" (UID: \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:43.529504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.529478    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b6bf862-nnpqd\" (UID: \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:43.529743 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.529723    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b6bf862-nnpqd\" (UID: \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:43.529948 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.529932    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38b34cf-4981-4932-82f8-b3d2856a5c00-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:43.530059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.530039    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pf2c\" (UniqueName: \"kubernetes.io/projected/d38b34cf-4981-4932-82f8-b3d2856a5c00-kube-api-access-9pf2c\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:43.530597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.530570    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b6bf862-nnpqd\" (UID: \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:43.533294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.533276    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b6bf862-nnpqd\" (UID: \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:43.607332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.607180    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5s45\" (UniqueName: \"kubernetes.io/projected/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-kube-api-access-p5s45\") pod \"maas-api-revoke-sub-9b6bf862-nnpqd\" (UID: \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:43.729543 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.729508    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:43.907484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.906961    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv" event={"ID":"d38b34cf-4981-4932-82f8-b3d2856a5c00","Type":"ContainerDied","Data":"eaf1a62658851bce948ac4fb5dc4d28fb504e80dce02a47b1e245be173c8add3"}
Nov 02 15:18:43.907484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.907026    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaf1a62658851bce948ac4fb5dc4d28fb504e80dce02a47b1e245be173c8add3"
Nov 02 15:18:43.907484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.907110    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"
Nov 02 15:18:43.925117 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.923745    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq" event={"ID":"a555e6ba-dc5d-45ac-89da-ed00a0205f65","Type":"ContainerStarted","Data":"7e0341744c87ca93f82e815a1e0b5687f50c643cb9a4e01fc7199792be3659d0"}
Nov 02 15:18:43.925117 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.923819    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq" event={"ID":"a555e6ba-dc5d-45ac-89da-ed00a0205f65","Type":"ContainerStarted","Data":"3794585dfae68dce306e29df9d67ed3fadee0e9ce8bba9aa6bc7c0e76f56a398"}
Nov 02 15:18:43.934255 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.933281    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv" event={"ID":"bebebeb8-f4ed-47d6-a491-ceddc58fc949","Type":"ContainerStarted","Data":"96f00358ee187cdb5c308e5a23bc0b4f8abd9132bf103e8fe64c4d632e52738f"}
Nov 02 15:18:43.934255 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.933378    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv" event={"ID":"bebebeb8-f4ed-47d6-a491-ceddc58fc949","Type":"ContainerStarted","Data":"d1a5bd5af261da2474c45f7916eac85ba0a4b55ea0a72e8668596520a196c93f"}
Nov 02 15:18:43.934792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:43.934731    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"
Nov 02 15:18:44.042509 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:44.042449    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq" podStartSLOduration=2.042435951 podStartE2EDuration="2.042435951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:44.040384944 +0000 UTC m=+1331.633001750" watchObservedRunningTime="2026-11-02 15:18:44.042435951 +0000 UTC m=+1331.635052775"
Nov 02 15:18:44.052869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:44.052829    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 15:18:44.053064 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:44.052906    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt" podUID="dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 15:18:44.071275 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:44.071240    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 15:18:44.071510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:44.071302    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7" podUID="1559e94a-36ed-40a4-8118-3d8f0f338a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 15:18:44.161542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:44.161477    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv" podStartSLOduration=3.161456855 podStartE2EDuration="3.161456855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:44.158961531 +0000 UTC m=+1331.751578331" watchObservedRunningTime="2026-11-02 15:18:44.161456855 +0000 UTC m=+1331.754073682"
Nov 02 15:18:44.211645 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:18:44.211612    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef40a5ee_b4e0_4a7a_bdeb_2f13de573d4b.slice/crio-1a4eda04c77605aae9108b7152d1af45904fc84c2e5509293e2d14b554acd463 WatchSource:0}: Error finding container 1a4eda04c77605aae9108b7152d1af45904fc84c2e5509293e2d14b554acd463: Status 404 returned error can't find the container with id 1a4eda04c77605aae9108b7152d1af45904fc84c2e5509293e2d14b554acd463
Nov 02 15:18:44.216268 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:44.216219    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"]
Nov 02 15:18:44.280525 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:18:44.280490    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:18:44.978937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:44.978829    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd" event={"ID":"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b","Type":"ContainerStarted","Data":"9d53a45be22c572f052eef4ea7fb86d3bfbf1c0019833b4ba01abe7784689e5a"}
Nov 02 15:18:44.979117 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:44.978939    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd" event={"ID":"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b","Type":"ContainerStarted","Data":"1a4eda04c77605aae9108b7152d1af45904fc84c2e5509293e2d14b554acd463"}
Nov 02 15:18:44.999925 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:44.999884    4891 generic.go:356] "Generic (PLEG): container finished" podID="a555e6ba-dc5d-45ac-89da-ed00a0205f65" containerID="7e0341744c87ca93f82e815a1e0b5687f50c643cb9a4e01fc7199792be3659d0" exitCode=0
Nov 02 15:18:45.000375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:45.000300    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq" event={"ID":"a555e6ba-dc5d-45ac-89da-ed00a0205f65","Type":"ContainerDied","Data":"7e0341744c87ca93f82e815a1e0b5687f50c643cb9a4e01fc7199792be3659d0"}
Nov 02 15:18:45.101878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:45.101809    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd" podStartSLOduration=2.101775389 podStartE2EDuration="2.101775389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:45.095685935 +0000 UTC m=+1332.688302736" watchObservedRunningTime="2026-11-02 15:18:45.101775389 +0000 UTC m=+1332.694392193"
Nov 02 15:18:45.347245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:45.347214    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:45.347837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:45.347803    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:45.348558 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:45.348536    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:18:45.356329 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:45.356286    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:45.356728 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:45.356702    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:45.357486 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:45.357470    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:18:46.010525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:46.010489    4891 generic.go:356] "Generic (PLEG): container finished" podID="ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b" containerID="9d53a45be22c572f052eef4ea7fb86d3bfbf1c0019833b4ba01abe7784689e5a" exitCode=0
Nov 02 15:18:46.010706 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:46.010575    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd" event={"ID":"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b","Type":"ContainerDied","Data":"9d53a45be22c572f052eef4ea7fb86d3bfbf1c0019833b4ba01abe7784689e5a"}
Nov 02 15:18:46.185684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:46.185660    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:46.306955 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:46.306924    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-openshift-ca\") pod \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\" (UID: \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\") "
Nov 02 15:18:46.307148 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:46.306969    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-xks-ca\") pod \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\" (UID: \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\") "
Nov 02 15:18:46.307148 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:46.307103    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a555e6ba-dc5d-45ac-89da-ed00a0205f65-kube-api-access-j6mzw\" (UniqueName: \"kubernetes.io/projected/a555e6ba-dc5d-45ac-89da-ed00a0205f65-kube-api-access-j6mzw\") pod \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\" (UID: \"a555e6ba-dc5d-45ac-89da-ed00a0205f65\") "
Nov 02 15:18:46.307379 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:46.307352    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-openshift-ca" pod "a555e6ba-dc5d-45ac-89da-ed00a0205f65" (UID: "a555e6ba-dc5d-45ac-89da-ed00a0205f65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:46.307937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:46.307909    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:46.309209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:46.309174    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-xks-ca" pod "a555e6ba-dc5d-45ac-89da-ed00a0205f65" (UID: "a555e6ba-dc5d-45ac-89da-ed00a0205f65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:46.309354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:46.309263    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a555e6ba-dc5d-45ac-89da-ed00a0205f65-kube-api-access-j6mzw" pod "a555e6ba-dc5d-45ac-89da-ed00a0205f65" (UID: "a555e6ba-dc5d-45ac-89da-ed00a0205f65"). InnerVolumeSpecName "kube-api-access-j6mzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:46.409247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:46.409210    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6mzw\" (UniqueName: \"kubernetes.io/projected/a555e6ba-dc5d-45ac-89da-ed00a0205f65-kube-api-access-j6mzw\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:46.409247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:46.409242    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a555e6ba-dc5d-45ac-89da-ed00a0205f65-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:47.022206 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.022178    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"
Nov 02 15:18:47.022427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.022209    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq" event={"ID":"a555e6ba-dc5d-45ac-89da-ed00a0205f65","Type":"ContainerDied","Data":"3794585dfae68dce306e29df9d67ed3fadee0e9ce8bba9aa6bc7c0e76f56a398"}
Nov 02 15:18:47.022427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.022263    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3794585dfae68dce306e29df9d67ed3fadee0e9ce8bba9aa6bc7c0e76f56a398"
Nov 02 15:18:47.202972 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.202943    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:47.323976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.323875    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-openshift-ca\") pod \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\" (UID: \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\") "
Nov 02 15:18:47.324146 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.324116    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-xks-ca\") pod \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\" (UID: \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\") "
Nov 02 15:18:47.324476 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.324453    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-kube-api-access-p5s45\" (UniqueName: \"kubernetes.io/projected/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-kube-api-access-p5s45\") pod \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\" (UID: \"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b\") "
Nov 02 15:18:47.324612 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.324509    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-openshift-ca" pod "ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b" (UID: "ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:47.325137 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.325111    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:47.328605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.328577    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-kube-api-access-p5s45" pod "ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b" (UID: "ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b"). InnerVolumeSpecName "kube-api-access-p5s45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:47.328729 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.328655    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-xks-ca" pod "ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b" (UID: "ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:47.426619 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.426583    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:47.426619 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:47.426616    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5s45\" (UniqueName: \"kubernetes.io/projected/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b-kube-api-access-p5s45\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:48.034178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.034147    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"
Nov 02 15:18:48.034178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.034160    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd" event={"ID":"ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b","Type":"ContainerDied","Data":"1a4eda04c77605aae9108b7152d1af45904fc84c2e5509293e2d14b554acd463"}
Nov 02 15:18:48.034465 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.034207    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a4eda04c77605aae9108b7152d1af45904fc84c2e5509293e2d14b554acd463"
Nov 02 15:18:48.672704 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.672663    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"]
Nov 02 15:18:48.673634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.673613    4891 cpu_manager.go:414] "containerMap: removing container" podUID="d38b34cf-4981-4932-82f8-b3d2856a5c00" containerName="revoke-keys"
Nov 02 15:18:48.673697 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.673637    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38b34cf-4981-4932-82f8-b3d2856a5c00" containerName="revoke-keys"
Nov 02 15:18:48.673697 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.673672    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b" containerName="revoke-keys"
Nov 02 15:18:48.673697 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.673681    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b" containerName="revoke-keys"
Nov 02 15:18:48.673783 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.673708    4891 cpu_manager.go:414] "containerMap: removing container" podUID="a555e6ba-dc5d-45ac-89da-ed00a0205f65" containerName="revoke-keys"
Nov 02 15:18:48.673783 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.673715    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a555e6ba-dc5d-45ac-89da-ed00a0205f65" containerName="revoke-keys"
Nov 02 15:18:48.673904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.673893    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38b34cf-4981-4932-82f8-b3d2856a5c00" containerName="revoke-keys"
Nov 02 15:18:48.673936 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.673912    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b" containerName="revoke-keys"
Nov 02 15:18:48.673936 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.673926    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="a555e6ba-dc5d-45ac-89da-ed00a0205f65" containerName="revoke-keys"
Nov 02 15:18:48.678541 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.678505    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:48.742338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.731591    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:18:48.742338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.732388    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"]
Nov 02 15:18:48.746746 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.746707    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8f1da9b-nvpfv\" (UID: \"fff03e96-2b02-46cf-b634-2f7acca977b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:48.747008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.746974    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8f1da9b-nvpfv\" (UID: \"fff03e96-2b02-46cf-b634-2f7acca977b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:48.747211 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.747108    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clnp4\" (UniqueName: \"kubernetes.io/projected/fff03e96-2b02-46cf-b634-2f7acca977b5-kube-api-access-clnp4\") pod \"maas-api-revoke-sub-d8f1da9b-nvpfv\" (UID: \"fff03e96-2b02-46cf-b634-2f7acca977b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:48.774615 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.774558    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:18:48.774615 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.774558    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:18:48.775001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.774984    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:18:48.848597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.848558    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clnp4\" (UniqueName: \"kubernetes.io/projected/fff03e96-2b02-46cf-b634-2f7acca977b5-kube-api-access-clnp4\") pod \"maas-api-revoke-sub-d8f1da9b-nvpfv\" (UID: \"fff03e96-2b02-46cf-b634-2f7acca977b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:48.848781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.848674    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8f1da9b-nvpfv\" (UID: \"fff03e96-2b02-46cf-b634-2f7acca977b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:48.848875 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.848848    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8f1da9b-nvpfv\" (UID: \"fff03e96-2b02-46cf-b634-2f7acca977b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:48.849535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.849510    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8f1da9b-nvpfv\" (UID: \"fff03e96-2b02-46cf-b634-2f7acca977b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:48.851146 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.851125    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8f1da9b-nvpfv\" (UID: \"fff03e96-2b02-46cf-b634-2f7acca977b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:48.910569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.910537    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clnp4\" (UniqueName: \"kubernetes.io/projected/fff03e96-2b02-46cf-b634-2f7acca977b5-kube-api-access-clnp4\") pod \"maas-api-revoke-sub-d8f1da9b-nvpfv\" (UID: \"fff03e96-2b02-46cf-b634-2f7acca977b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:48.993293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:48.993210    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:49.385618 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:18:49.385587    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff03e96_2b02_46cf_b634_2f7acca977b5.slice/crio-faf01ab52df317063f293dffb25606a5e1d118deb58f50e9ef2933bda4ce9c24 WatchSource:0}: Error finding container faf01ab52df317063f293dffb25606a5e1d118deb58f50e9ef2933bda4ce9c24: Status 404 returned error can't find the container with id faf01ab52df317063f293dffb25606a5e1d118deb58f50e9ef2933bda4ce9c24
Nov 02 15:18:49.388837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:49.388810    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"]
Nov 02 15:18:50.059083 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:50.059045    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv" event={"ID":"fff03e96-2b02-46cf-b634-2f7acca977b5","Type":"ContainerStarted","Data":"3d5df48263216c255554688df21e881655fb35334a271eb97c271a3d7dc80100"}
Nov 02 15:18:50.059586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:50.059108    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv" event={"ID":"fff03e96-2b02-46cf-b634-2f7acca977b5","Type":"ContainerStarted","Data":"faf01ab52df317063f293dffb25606a5e1d118deb58f50e9ef2933bda4ce9c24"}
Nov 02 15:18:50.180264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:50.180198    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv" podStartSLOduration=2.180179277 podStartE2EDuration="2.180179277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:50.177592317 +0000 UTC m=+1337.770209115" watchObservedRunningTime="2026-11-02 15:18:50.180179277 +0000 UTC m=+1337.772796087"
Nov 02 15:18:51.105558 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:51.104743    4891 generic.go:356] "Generic (PLEG): container finished" podID="fff03e96-2b02-46cf-b634-2f7acca977b5" containerID="3d5df48263216c255554688df21e881655fb35334a271eb97c271a3d7dc80100" exitCode=0
Nov 02 15:18:51.105558 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:51.105100    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv" event={"ID":"fff03e96-2b02-46cf-b634-2f7acca977b5","Type":"ContainerDied","Data":"3d5df48263216c255554688df21e881655fb35334a271eb97c271a3d7dc80100"}
Nov 02 15:18:51.952172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:51.950456    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:51.952172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:51.951494    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:52.211389 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:52.204742    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:52.433120 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:52.433075    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:52.434601 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:52.434569    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:18:52.478625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:52.478530    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:52.614993 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:52.614956    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:52.629354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:52.628501    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded"
Nov 02 15:18:52.629659 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:52.629628    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:52.899336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:52.897419    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:52.933192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:52.916621    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:52.953336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:52.951519    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:52.953336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:52.951763    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:52.953336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:52.953176    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:18:53.019351 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.006433    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:53.118940 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.113423    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:53.118940 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.113652    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:18:53.129267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.129222    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:18:53.230395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.227487    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:18:53.230395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.228088    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:18:53.397350 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.258965    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 15:18:53.550395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.500401    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 15:18:53.550395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.438413    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:18:53.550395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.501125    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:18:53.550395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.501186    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:18:53.550395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.438446    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 15:18:53.550395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.503167    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:18:53.550395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.289410    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:53.550395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.504995    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded"
Nov 02 15:18:53.624873 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.617658    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 15:18:53.667337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.645532    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:53.671768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.671736    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:18:53.697489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.697450    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:18:53.698673 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.647599    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:53.899838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.899783    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 15:18:53.928337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.903779    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:18:53.992478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.992428    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 15:18:53.993185 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:53.993162    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:18:54.019346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.015610    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 02 15:18:54.019346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.016060    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:18:54.019346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.016967    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Nov 02 15:18:54.045381 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.045339    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:54.082337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.082268    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:54.094340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.087151    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:18:54.120630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.115847    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:54.120630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.116266    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 02 15:18:54.120630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.116714    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:18:54.120630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.116793    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:18:54.120630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.118900    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:18:54.120630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.118958    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:54.193655 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.193612    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:54.194297 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.194272    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:18:54.286336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.258300    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:18:54.488597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.488544    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:54.489261 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.489234    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:18:54.504386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.504350    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 15:18:54.505339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.505285    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 15:18:54.505789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.505763    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:18:54.715394 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.712662    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:54.715394 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.714380    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:54.966387 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.950147    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:54.984585 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.984427    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:18:54.984585 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:54.984506    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:55.041339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.040288    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:55.071687 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.067605    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:55.364277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.071950    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:55.364277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.087571    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:18:55.364277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.117577    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:55.364277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.117610    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:18:55.364277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.288780    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:55.364277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.318668    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:18:55.403990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.403941    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:55.437711 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.437659    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:55.485005 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.484975    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-openshift-ca\") pod \"fff03e96-2b02-46cf-b634-2f7acca977b5\" (UID: \"fff03e96-2b02-46cf-b634-2f7acca977b5\") "
Nov 02 15:18:55.512539 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.512478    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 15:18:55.513233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.513193    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:55.513753 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.513722    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 15:18:55.514183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.514162    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 15:18:55.541457 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.541383    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-openshift-ca" pod "fff03e96-2b02-46cf-b634-2f7acca977b5" (UID: "fff03e96-2b02-46cf-b634-2f7acca977b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:55.677416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.661071    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-xks-ca\") pod \"fff03e96-2b02-46cf-b634-2f7acca977b5\" (UID: \"fff03e96-2b02-46cf-b634-2f7acca977b5\") "
Nov 02 15:18:55.677416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.661149    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff03e96-2b02-46cf-b634-2f7acca977b5-kube-api-access-clnp4\" (UniqueName: \"kubernetes.io/projected/fff03e96-2b02-46cf-b634-2f7acca977b5-kube-api-access-clnp4\") pod \"fff03e96-2b02-46cf-b634-2f7acca977b5\" (UID: \"fff03e96-2b02-46cf-b634-2f7acca977b5\") "
Nov 02 15:18:55.718347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.717921    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:18:55.718347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.718339    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:55.780332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.779635    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv" event={"ID":"fff03e96-2b02-46cf-b634-2f7acca977b5","Type":"ContainerDied","Data":"faf01ab52df317063f293dffb25606a5e1d118deb58f50e9ef2933bda4ce9c24"}
Nov 02 15:18:55.780332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.779735    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faf01ab52df317063f293dffb25606a5e1d118deb58f50e9ef2933bda4ce9c24"
Nov 02 15:18:55.780332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.779813    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"
Nov 02 15:18:55.803116 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.803085    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:55.839570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.839533    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff03e96-2b02-46cf-b634-2f7acca977b5-kube-api-access-clnp4" pod "fff03e96-2b02-46cf-b634-2f7acca977b5" (UID: "fff03e96-2b02-46cf-b634-2f7acca977b5"). InnerVolumeSpecName "kube-api-access-clnp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:55.918973 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.918936    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clnp4\" (UniqueName: \"kubernetes.io/projected/fff03e96-2b02-46cf-b634-2f7acca977b5-kube-api-access-clnp4\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:55.919300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.919272    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-xks-ca" pod "fff03e96-2b02-46cf-b634-2f7acca977b5" (UID: "fff03e96-2b02-46cf-b634-2f7acca977b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:55.971499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:55.971394    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:56.076332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.027298    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff03e96-2b02-46cf-b634-2f7acca977b5-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:18:56.094435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.058720    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:56.377517 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.363550    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:18:56.378192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.363584    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 15:18:56.389402 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.363602    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:18:56.448043 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.363602    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:56.456121 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.363619    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:18:56.456121 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.363636    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:18:56.456121 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.363711    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:56.522345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.520719    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:56.522547 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.522373    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:56.533345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.523422    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:56.533345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.523536    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 15:18:56.732325 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.732189    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:56.854035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.732837    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:56.854035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:56.837291    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:57.111346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.082860    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"
Nov 02 15:18:57.111715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.088665    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:57.111921 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.090707    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:57.380510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.380400    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 02 15:18:57.433127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.394386    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 15:18:57.433127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.394772    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 02 15:18:57.559958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.444122    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:18:57.559958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.444745    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:57.559958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.445052    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:18:57.559958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.450589    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:57.559958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.463420    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 15:18:57.559958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.492463    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:18:57.580324 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.580268    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:57.617843 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.617798    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:57.622815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.622776    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:57.627140 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.625405    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:57.769236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.767423    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:18:57.769236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.768667    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 15:18:57.826960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.826931    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:18:57.901168 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.901137    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:18:57.952455 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.952420    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:18:57.979154 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:57.979122    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:18:58.003348 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:58.003275    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:18:58.004183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:58.004163    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:18:58.047538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:58.045813    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:18:58.091613 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:58.090736    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:18:58.091613 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:58.091243    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:18:58.106926 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:58.106905    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:18:58.116472 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:58.116448    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:18:58.123499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:58.123478    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:18:58.139539 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:58.139513    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:18:58.172965 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:18:58.172939    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:19:14.274594 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:19:14.274553    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:19:22.000643 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.000609    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"]
Nov 02 15:19:22.001462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.001436    4891 cpu_manager.go:414] "containerMap: removing container" podUID="fff03e96-2b02-46cf-b634-2f7acca977b5" containerName="revoke-keys"
Nov 02 15:19:22.001462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.001462    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff03e96-2b02-46cf-b634-2f7acca977b5" containerName="revoke-keys"
Nov 02 15:19:22.001643 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.001627    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff03e96-2b02-46cf-b634-2f7acca977b5" containerName="revoke-keys"
Nov 02 15:19:22.005458 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.005437    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.067570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.067536    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"]
Nov 02 15:19:22.089731 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.089695    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.089906 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.089830    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a3b4a5-282a-473b-8d71-287a475429b1-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.089971 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.089926    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.090060 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.090041    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.090165 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.090081    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.090242 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.090226    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.090371 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.090355    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.090582 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.090565    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.090679 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.090637    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knrrr\" (UniqueName: \"kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-kube-api-access-knrrr\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.192206 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.192168    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.192455 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.192276    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.192455 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.192375    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.192455 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.192424    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knrrr\" (UniqueName: \"kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-kube-api-access-knrrr\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.192635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.192497    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.192635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.192579    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a3b4a5-282a-473b-8d71-287a475429b1-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.192742 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.192671    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.192742 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.192679    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-credential-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.192848 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.192739    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.192848 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.192772    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.192952 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.192864    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-data\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.193069 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.193047    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-socket\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.193135 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.193105    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-certs\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.193435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.193415    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a3b4a5-282a-473b-8d71-287a475429b1-istiod-ca-cert\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.194607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.194582    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-envoy\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.194974 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.194953    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-podinfo\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.253561 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.253493    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-token\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.253677 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.253586    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knrrr\" (UniqueName: \"kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-kube-api-access-knrrr\") pod \"e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") " pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.320020 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.319988    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:22.789997 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:19:22.789965    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a3b4a5_282a_473b_8d71_287a475429b1.slice/crio-4b3fab7c4912219fbb90652988f3bd0170b9134a2d7173b910d5934cc5b81ece WatchSource:0}: Error finding container 4b3fab7c4912219fbb90652988f3bd0170b9134a2d7173b910d5934cc5b81ece: Status 404 returned error can't find the container with id 4b3fab7c4912219fbb90652988f3bd0170b9134a2d7173b910d5934cc5b81ece
Nov 02 15:19:22.792860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.792830    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"]
Nov 02 15:19:22.793162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.793123    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:22.793275 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.793195    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:22.793275 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:22.793237    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:23.047542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:23.047454    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" event={"ID":"d2a3b4a5-282a-473b-8d71-287a475429b1","Type":"ContainerStarted","Data":"69e1b4d604670ad26bb4a634ab5fab935a414304d303a771410e68878e640010"}
Nov 02 15:19:23.047542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:23.047520    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" event={"ID":"d2a3b4a5-282a-473b-8d71-287a475429b1","Type":"ContainerStarted","Data":"4b3fab7c4912219fbb90652988f3bd0170b9134a2d7173b910d5934cc5b81ece"}
Nov 02 15:19:23.170295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:23.170226    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" podStartSLOduration=2.17020735 podStartE2EDuration="2.17020735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:23.167324146 +0000 UTC m=+1370.759940941" watchObservedRunningTime="2026-11-02 15:19:23.17020735 +0000 UTC m=+1370.762824150"
Nov 02 15:19:23.321239 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:23.321164    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:23.322689 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:23.322662    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:19:23.322813 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:23.322712    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" podUID="d2a3b4a5-282a-473b-8d71-287a475429b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:19:24.321415 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:24.321375    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:19:24.321822 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:24.321437    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" podUID="d2a3b4a5-282a-473b-8d71-287a475429b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:19:25.320348 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:25.320297    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:19:25.320533 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:25.320366    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" podUID="d2a3b4a5-282a-473b-8d71-287a475429b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:19:26.321035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:26.320997    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:19:26.321434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:26.321054    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" podUID="d2a3b4a5-282a-473b-8d71-287a475429b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:19:27.320529 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:27.320488    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:19:27.320716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:27.320546    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" podUID="d2a3b4a5-282a-473b-8d71-287a475429b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:19:28.321116 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:28.321079    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:19:28.321583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:28.321136    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" podUID="d2a3b4a5-282a-473b-8d71-287a475429b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:19:29.320788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:29.320746    4891 patch_prober.go:29] interesting pod/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 15:19:29.320989 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:29.320809    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" podUID="d2a3b4a5-282a-473b-8d71-287a475429b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 15:19:30.326162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:30.326131    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:30.326695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:30.326675    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:30.327435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:30.327413    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:19:33.133335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:33.132817    4891 scope.go:122] "RemoveContainer" containerID="dfbf858181ce7174e4cb2a0eb213afdc02b5f4afbe83ed0e719d39ae008b6493"
Nov 02 15:19:33.298723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:33.298694    4891 scope.go:122] "RemoveContainer" containerID="bfd328287ba990ef091c29e55a9d1e9b239e655428a2af764b425b50f149132b"
Nov 02 15:19:33.503793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:33.501498    4891 scope.go:122] "RemoveContainer" containerID="a5b33178f3b05e3e7b8098d47bb4a7e53253455488dadf8236c9777144518ed5"
Nov 02 15:19:33.510297 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:33.509216    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 15:19:33.531444 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:33.531405    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 15:19:33.577514 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:33.577478    4891 scope.go:122] "RemoveContainer" containerID="3f580c12c5917a63c594ae7f2530db8d9eac57d30f7208d4a57ed8d8bdef6676"
Nov 02 15:19:33.641828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:33.641689    4891 scope.go:122] "RemoveContainer" containerID="f6456c4c14c47d73ddf2044e47858d74d0c42e598e6dbb04196e7ef5be6fc6cc"
Nov 02 15:19:33.716228 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:33.713290    4891 scope.go:122] "RemoveContainer" containerID="cdf5f6b4eaaabaf7426dc5fd591d5f7a96fa2510d46e7b6584976ab86186ffc0"
Nov 02 15:19:33.748066 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:33.745677    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:19:44.275024 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:19:44.274936    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:19:53.679072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.679037    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"]
Nov 02 15:19:53.683550 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.683530    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.744544 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.744511    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.744544 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.744545    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8s97\" (UniqueName: \"kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-kube-api-access-r8s97\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.744746 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.744679    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6483392d-cba3-48d0-a898-73d7369d3e20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.744746 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.744710    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.744812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.744786    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.744812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.744803    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.744904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.744882    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6483392d-cba3-48d0-a898-73d7369d3e20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.744953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.744941    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.745056 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.745039    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.786489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.786457    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"]
Nov 02 15:19:53.846294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.846259    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.846294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.846293    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8s97\" (UniqueName: \"kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-kube-api-access-r8s97\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.846589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.846378    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6483392d-cba3-48d0-a898-73d7369d3e20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.846589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.846406    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.846589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.846455    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.846589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.846470    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.846589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.846503    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6483392d-cba3-48d0-a898-73d7369d3e20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.846589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.846565    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.846889 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.846630    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.846889 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.846737    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.846889 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.846877    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.847049 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.846969    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.847049 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.847017    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.847160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.847120    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6483392d-cba3-48d0-a898-73d7369d3e20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.848809 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.848783    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.848995 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.848976    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6483392d-cba3-48d0-a898-73d7369d3e20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.909284 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.909247    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8s97\" (UniqueName: \"kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-kube-api-access-r8s97\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.910636 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.910616    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lcdkf\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:53.998373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:53.998259    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:54.438617 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:19:54.438585    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6483392d_cba3_48d0_a898_73d7369d3e20.slice/crio-f45975bad25426174ed78ba7bfac80c0990104186c322b0fdb0587e5fbcef8ea WatchSource:0}: Error finding container f45975bad25426174ed78ba7bfac80c0990104186c322b0fdb0587e5fbcef8ea: Status 404 returned error can't find the container with id f45975bad25426174ed78ba7bfac80c0990104186c322b0fdb0587e5fbcef8ea
Nov 02 15:19:54.439856 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:54.439835    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"]
Nov 02 15:19:54.441638 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:54.441606    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:54.441703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:54.441669    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:54.441703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:54.441699    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:54.539237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:54.539201    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" event={"ID":"6483392d-cba3-48d0-a898-73d7369d3e20","Type":"ContainerStarted","Data":"3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669"}
Nov 02 15:19:54.539422 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:54.539267    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" event={"ID":"6483392d-cba3-48d0-a898-73d7369d3e20","Type":"ContainerStarted","Data":"f45975bad25426174ed78ba7bfac80c0990104186c322b0fdb0587e5fbcef8ea"}
Nov 02 15:19:54.662258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:54.662202    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" podStartSLOduration=1.662187899 podStartE2EDuration="1.662187899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:54.659956808 +0000 UTC m=+1402.252573644" watchObservedRunningTime="2026-11-02 15:19:54.662187899 +0000 UTC m=+1402.254804698"
Nov 02 15:19:54.998993 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:54.998958    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:19:56.000701 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:56.000664    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:19:56.001110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:56.000720    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:19:56.999071 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:56.999034    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:19:56.999266 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:56.999094    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:19:57.998778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:57.998744    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:19:57.999203 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:57.998794    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:19:58.998665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:58.998630    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:19:58.998844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:58.998686    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:19:59.998863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:59.998822    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:19:59.999274 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:19:59.998890    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:20:00.999399 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:00.999363    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:20:00.999817 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:00.999426    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:20:01.999126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:01.999088    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 15:20:01.999340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:01.999146    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 15:20:03.003966 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:03.003935    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:20:03.004434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:03.004409    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:20:03.005149 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:03.005128    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:20:08.836470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.836434    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"]
Nov 02 15:20:08.841090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.841072    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:08.850938 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.850915    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"]
Nov 02 15:20:08.856229 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.856210    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:08.898865 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.898832    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:08.899029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.898871    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:08.899029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.898897    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:08.899029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899014    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:08.899143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899052    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:08.899143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899126    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dc8955-8714-49cd-a3a3-001d46023fc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:08.899208 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899195    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwt29\" (UniqueName: \"kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-kube-api-access-wwt29\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:08.899244 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899227    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:08.899280 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899262    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:08.899352 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899291    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:08.899459 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899446    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:08.899499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899479    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:08.899536 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899508    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:08.899570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899547    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:08.899604 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899594    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:08.899648 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899632    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:08.899725 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899704    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:08.899841 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.899827    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhmq9\" (UniqueName: \"kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-kube-api-access-xhmq9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:08.918508 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.918480    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"]
Nov 02 15:20:08.982580 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:08.982548    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"]
Nov 02 15:20:09.001895 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.001859    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.002128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.001920    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.002128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.001990    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dc8955-8714-49cd-a3a3-001d46023fc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.002128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002046    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwt29\" (UniqueName: \"kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-kube-api-access-wwt29\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.002128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002094    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.002405 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002137    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.002405 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002162    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.002405 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002269    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.002405 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002326    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.002405 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002364    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.002405 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002399    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.002716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002458    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.002716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002480    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.002716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002479    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.002716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002508    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.002716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002581    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhmq9\" (UniqueName: \"kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-kube-api-access-xhmq9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.002716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002626    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.002716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002652    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.002716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002672    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.002716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002689    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.003179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002779    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.003179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.002888    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dc8955-8714-49cd-a3a3-001d46023fc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.003179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.003004    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.003179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.003055    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.003436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.003333    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.003604 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.003582    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.003689 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.003641    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.003844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.003823    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.006016 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.005985    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.006176 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.006114    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.007560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.007532    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.008099 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.008071    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.070174 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.070120    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.070462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.070440    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwt29\" (UniqueName: \"kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-kube-api-access-wwt29\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6qcng\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.070516 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.070452    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhmq9\" (UniqueName: \"kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-kube-api-access-xhmq9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.076155 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.076129    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zzbkt\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.158425 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.158345    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:09.169997 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.169971    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:09.600569 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:20:09.600528    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dc8955_8714_49cd_a3a3_001d46023fc3.slice/crio-e56336e3fd0bcddb1b1c2ecbdd2b8c32a0ba9c182f56abab407a7738d0b100d9 WatchSource:0}: Error finding container e56336e3fd0bcddb1b1c2ecbdd2b8c32a0ba9c182f56abab407a7738d0b100d9: Status 404 returned error can't find the container with id e56336e3fd0bcddb1b1c2ecbdd2b8c32a0ba9c182f56abab407a7738d0b100d9
Nov 02 15:20:09.602547 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:20:09.602522    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fb2ba3_1e31_4db1_9b63_94a48bff0663.slice/crio-267b45fda596d6b3ba1105c7f1dd404b03088fe9d401aceb3f8c92a83d9515f2 WatchSource:0}: Error finding container 267b45fda596d6b3ba1105c7f1dd404b03088fe9d401aceb3f8c92a83d9515f2: Status 404 returned error can't find the container with id 267b45fda596d6b3ba1105c7f1dd404b03088fe9d401aceb3f8c92a83d9515f2
Nov 02 15:20:09.602664 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.602556    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"]
Nov 02 15:20:09.603063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.603025    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:20:09.603140 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.603112    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:20:09.603200 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.603157    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:20:09.605386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.605355    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:20:09.605475 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.605422    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:20:09.605475 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.605451    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:20:09.609464 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.609447    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"]
Nov 02 15:20:09.709758 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.709707    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" event={"ID":"68fb2ba3-1e31-4db1-9b63-94a48bff0663","Type":"ContainerStarted","Data":"5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7"}
Nov 02 15:20:09.709758 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.709763    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" event={"ID":"68fb2ba3-1e31-4db1-9b63-94a48bff0663","Type":"ContainerStarted","Data":"267b45fda596d6b3ba1105c7f1dd404b03088fe9d401aceb3f8c92a83d9515f2"}
Nov 02 15:20:09.717133 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.716405    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" event={"ID":"b3dc8955-8714-49cd-a3a3-001d46023fc3","Type":"ContainerStarted","Data":"2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8"}
Nov 02 15:20:09.717133 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.716466    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" event={"ID":"b3dc8955-8714-49cd-a3a3-001d46023fc3","Type":"ContainerStarted","Data":"e56336e3fd0bcddb1b1c2ecbdd2b8c32a0ba9c182f56abab407a7738d0b100d9"}
Nov 02 15:20:09.834634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.834579    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" podStartSLOduration=1.834564133 podStartE2EDuration="1.834564133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:09.829803874 +0000 UTC m=+1417.422420674" watchObservedRunningTime="2026-11-02 15:20:09.834564133 +0000 UTC m=+1417.427181006"
Nov 02 15:20:09.962940 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:09.962813    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" podStartSLOduration=1.962794172 podStartE2EDuration="1.962794172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:09.960455921 +0000 UTC m=+1417.553072733" watchObservedRunningTime="2026-11-02 15:20:09.962794172 +0000 UTC m=+1417.555410985"
Nov 02 15:20:10.158788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:10.158753    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:10.160176 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:10.160148    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:20:10.160295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:10.160197    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 15:20:10.170331 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:10.170296    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:10.171592 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:10.171569    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6qcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 15:20:10.171690 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:10.171608    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 15:20:11.170335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:11.161700    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:20:11.170335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:11.161767    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 15:20:11.190340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:11.181306    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6qcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 15:20:11.190340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:11.181407    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 15:20:12.165038 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:12.165000    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:20:12.165335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:12.165287    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 15:20:12.200885 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:12.197079    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6qcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 15:20:12.200885 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:12.200426    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 15:20:13.164334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:13.161412    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:20:13.164334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:13.161477    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 15:20:13.177445 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:13.177406    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6qcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 15:20:13.177709 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:13.177681    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 15:20:14.162897 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:14.162847    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:20:14.163566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:14.163516    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 15:20:14.171761 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:14.171729    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6qcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 15:20:14.172028 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:14.172002    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 15:20:14.320519 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:20:14.320478    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:20:15.177462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:15.177418    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:20:15.178160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:15.178126    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 15:20:15.178756 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:15.178042    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6qcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 15:20:15.178924 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:15.178905    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 15:20:16.176775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:16.176733    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:20:16.177169 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:16.177135    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 15:20:16.177327 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:16.177076    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6qcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 15:20:16.177448 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:16.177341    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 15:20:17.159064 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:17.159019    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:20:17.159488 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:17.159093    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 15:20:17.170376 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:17.170299    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6qcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 15:20:17.170561 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:17.170404    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 15:20:18.159245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:18.159203    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:20:18.159694 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:18.159274    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 15:20:18.842977 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:18.842749    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:18.843400 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:18.843374    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:18.844402 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:18.844381    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:20:19.542342 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:19.541668    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:19.543554 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:19.543529    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:19.545161 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:19.545129    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:20:22.996009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:22.995964    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"]
Nov 02 15:20:23.001853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.001829    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:23.056568 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.056531    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:20:23.056772 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.056585    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:20:23.063380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.063346    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"]
Nov 02 15:20:23.103219 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.103178    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:20:23.103470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.103230    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9\" (UID: \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:23.103603 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.103558    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrkj\" (UniqueName: \"kubernetes.io/projected/64e14513-d6ca-4e5d-af48-fdfad82d8d45-kube-api-access-mdrkj\") pod \"maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9\" (UID: \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:23.103765 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.103724    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9\" (UID: \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:23.106107 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.106077    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:20:23.204768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.204736    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9\" (UID: \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:23.204945 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.204799    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9\" (UID: \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:23.204999 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.204988    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrkj\" (UniqueName: \"kubernetes.io/projected/64e14513-d6ca-4e5d-af48-fdfad82d8d45-kube-api-access-mdrkj\") pod \"maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9\" (UID: \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:23.205507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.205483    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9\" (UID: \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:23.207446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.207420    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9\" (UID: \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:23.267074 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.267004    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrkj\" (UniqueName: \"kubernetes.io/projected/64e14513-d6ca-4e5d-af48-fdfad82d8d45-kube-api-access-mdrkj\") pod \"maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9\" (UID: \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:23.318192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.318156    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:23.761060 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:20:23.761026    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e14513_d6ca_4e5d_af48_fdfad82d8d45.slice/crio-813ae3e24f080a22521fa524e415125a93e53c4b6a6ff2f0082ebe6e91ef8fbd WatchSource:0}: Error finding container 813ae3e24f080a22521fa524e415125a93e53c4b6a6ff2f0082ebe6e91ef8fbd: Status 404 returned error can't find the container with id 813ae3e24f080a22521fa524e415125a93e53c4b6a6ff2f0082ebe6e91ef8fbd
Nov 02 15:20:23.764862 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.764836    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"]
Nov 02 15:20:23.967334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:23.966441    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9" event={"ID":"64e14513-d6ca-4e5d-af48-fdfad82d8d45","Type":"ContainerStarted","Data":"813ae3e24f080a22521fa524e415125a93e53c4b6a6ff2f0082ebe6e91ef8fbd"}
Nov 02 15:20:24.433905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.433869    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"]
Nov 02 15:20:24.438888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.438852    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:24.496057 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.495948    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"]
Nov 02 15:20:24.542207 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.542170    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c20b67a2-q2d95\" (UID: \"995af847-3a60-4464-910f-d8d66dbc3672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:24.542536 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.542516    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjwx\" (UniqueName: \"kubernetes.io/projected/995af847-3a60-4464-910f-d8d66dbc3672-kube-api-access-tmjwx\") pod \"maas-api-revoke-sub-c20b67a2-q2d95\" (UID: \"995af847-3a60-4464-910f-d8d66dbc3672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:24.542673 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.542657    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c20b67a2-q2d95\" (UID: \"995af847-3a60-4464-910f-d8d66dbc3672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:24.644681 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.644640    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c20b67a2-q2d95\" (UID: \"995af847-3a60-4464-910f-d8d66dbc3672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:24.645063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.645035    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjwx\" (UniqueName: \"kubernetes.io/projected/995af847-3a60-4464-910f-d8d66dbc3672-kube-api-access-tmjwx\") pod \"maas-api-revoke-sub-c20b67a2-q2d95\" (UID: \"995af847-3a60-4464-910f-d8d66dbc3672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:24.645230 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.645136    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c20b67a2-q2d95\" (UID: \"995af847-3a60-4464-910f-d8d66dbc3672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:24.645703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.645675    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c20b67a2-q2d95\" (UID: \"995af847-3a60-4464-910f-d8d66dbc3672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:24.648239 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.648213    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c20b67a2-q2d95\" (UID: \"995af847-3a60-4464-910f-d8d66dbc3672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:24.706527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.706445    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjwx\" (UniqueName: \"kubernetes.io/projected/995af847-3a60-4464-910f-d8d66dbc3672-kube-api-access-tmjwx\") pod \"maas-api-revoke-sub-c20b67a2-q2d95\" (UID: \"995af847-3a60-4464-910f-d8d66dbc3672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:24.761921 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.761884    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:24.978989 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.978896    4891 generic.go:356] "Generic (PLEG): container finished" podID="64e14513-d6ca-4e5d-af48-fdfad82d8d45" containerID="7c1db10e74eedc1efa19da8464d1788260f7f6e62abeb0ffa196dee588e82e98" exitCode=0
Nov 02 15:20:24.979136 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:24.978977    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9" event={"ID":"64e14513-d6ca-4e5d-af48-fdfad82d8d45","Type":"ContainerDied","Data":"7c1db10e74eedc1efa19da8464d1788260f7f6e62abeb0ffa196dee588e82e98"}
Nov 02 15:20:25.160049 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:20:25.160018    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995af847_3a60_4464_910f_d8d66dbc3672.slice/crio-4283e80496e5466acf30074b9532b6872468339ba6f6726e33661f92a0dff184 WatchSource:0}: Error finding container 4283e80496e5466acf30074b9532b6872468339ba6f6726e33661f92a0dff184: Status 404 returned error can't find the container with id 4283e80496e5466acf30074b9532b6872468339ba6f6726e33661f92a0dff184
Nov 02 15:20:25.160168 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.160068    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"]
Nov 02 15:20:25.382889 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.382856    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"]
Nov 02 15:20:25.389924 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.389893    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:25.454346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.454297    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"]
Nov 02 15:20:25.560061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.560033    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk\" (UID: \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:25.560224 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.560094    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwwcw\" (UniqueName: \"kubernetes.io/projected/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-kube-api-access-jwwcw\") pod \"maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk\" (UID: \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:25.560504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.560481    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk\" (UID: \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:25.662351 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.662150    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk\" (UID: \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:25.662351 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.662305    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk\" (UID: \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:25.662611 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.662416    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwwcw\" (UniqueName: \"kubernetes.io/projected/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-kube-api-access-jwwcw\") pod \"maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk\" (UID: \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:25.663242 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.663215    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk\" (UID: \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:25.664732 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.664709    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk\" (UID: \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:25.731548 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.731518    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwwcw\" (UniqueName: \"kubernetes.io/projected/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-kube-api-access-jwwcw\") pod \"maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk\" (UID: \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:25.991545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.991448    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95" event={"ID":"995af847-3a60-4464-910f-d8d66dbc3672","Type":"ContainerStarted","Data":"d0845fe43f4f8b84dc43d1990f795bc63f00b28cfa104767ea1d0b33178c73c9"}
Nov 02 15:20:25.991545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:25.991519    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95" event={"ID":"995af847-3a60-4464-910f-d8d66dbc3672","Type":"ContainerStarted","Data":"4283e80496e5466acf30074b9532b6872468339ba6f6726e33661f92a0dff184"}
Nov 02 15:20:26.007552 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.007521    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:26.116928 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.116866    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95" podStartSLOduration=2.116846177 podStartE2EDuration="2.116846177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:26.107071804 +0000 UTC m=+1433.699688607" watchObservedRunningTime="2026-11-02 15:20:26.116846177 +0000 UTC m=+1433.709462978"
Nov 02 15:20:26.177684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.177277    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:26.272409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.272378    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-xks-ca\") pod \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\" (UID: \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\") "
Nov 02 15:20:26.272583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.272564    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-openshift-ca\") pod \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\" (UID: \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\") "
Nov 02 15:20:26.272648 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.272601    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e14513-d6ca-4e5d-af48-fdfad82d8d45-kube-api-access-mdrkj\" (UniqueName: \"kubernetes.io/projected/64e14513-d6ca-4e5d-af48-fdfad82d8d45-kube-api-access-mdrkj\") pod \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\" (UID: \"64e14513-d6ca-4e5d-af48-fdfad82d8d45\") "
Nov 02 15:20:26.272958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.272935    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-openshift-ca" pod "64e14513-d6ca-4e5d-af48-fdfad82d8d45" (UID: "64e14513-d6ca-4e5d-af48-fdfad82d8d45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:26.273411 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.273384    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:26.275039 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.275018    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-xks-ca" pod "64e14513-d6ca-4e5d-af48-fdfad82d8d45" (UID: "64e14513-d6ca-4e5d-af48-fdfad82d8d45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:20:26.275127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.275020    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e14513-d6ca-4e5d-af48-fdfad82d8d45-kube-api-access-mdrkj" pod "64e14513-d6ca-4e5d-af48-fdfad82d8d45" (UID: "64e14513-d6ca-4e5d-af48-fdfad82d8d45"). InnerVolumeSpecName "kube-api-access-mdrkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:26.375033 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.374983    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e14513-d6ca-4e5d-af48-fdfad82d8d45-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:26.375033 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.375030    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdrkj\" (UniqueName: \"kubernetes.io/projected/64e14513-d6ca-4e5d-af48-fdfad82d8d45-kube-api-access-mdrkj\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:26.443553 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:20:26.443507    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18ef3b4_6a05_46ae_8a34_9c7535359b6b.slice/crio-4f69eea623f2019fd8c2796faa1d18d51f83989639f717e0d8a0f1abd201eddc WatchSource:0}: Error finding container 4f69eea623f2019fd8c2796faa1d18d51f83989639f717e0d8a0f1abd201eddc: Status 404 returned error can't find the container with id 4f69eea623f2019fd8c2796faa1d18d51f83989639f717e0d8a0f1abd201eddc
Nov 02 15:20:26.444935 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:26.444909    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"]
Nov 02 15:20:27.002781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:27.002752    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"
Nov 02 15:20:27.003193 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:27.002778    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9" event={"ID":"64e14513-d6ca-4e5d-af48-fdfad82d8d45","Type":"ContainerDied","Data":"813ae3e24f080a22521fa524e415125a93e53c4b6a6ff2f0082ebe6e91ef8fbd"}
Nov 02 15:20:27.003193 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:27.002828    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="813ae3e24f080a22521fa524e415125a93e53c4b6a6ff2f0082ebe6e91ef8fbd"
Nov 02 15:20:27.005205 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:27.004836    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk" event={"ID":"b18ef3b4-6a05-46ae-8a34-9c7535359b6b","Type":"ContainerStarted","Data":"e0bc94bada53baa514935114fde5dd5caae6ce25601fd16ca0004854c62107fe"}
Nov 02 15:20:27.005441 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:27.005411    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk" event={"ID":"b18ef3b4-6a05-46ae-8a34-9c7535359b6b","Type":"ContainerStarted","Data":"4f69eea623f2019fd8c2796faa1d18d51f83989639f717e0d8a0f1abd201eddc"}
Nov 02 15:20:27.006732 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:27.006702    4891 generic.go:356] "Generic (PLEG): container finished" podID="995af847-3a60-4464-910f-d8d66dbc3672" containerID="d0845fe43f4f8b84dc43d1990f795bc63f00b28cfa104767ea1d0b33178c73c9" exitCode=0
Nov 02 15:20:27.006820 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:27.006771    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95" event={"ID":"995af847-3a60-4464-910f-d8d66dbc3672","Type":"ContainerDied","Data":"d0845fe43f4f8b84dc43d1990f795bc63f00b28cfa104767ea1d0b33178c73c9"}
Nov 02 15:20:27.128693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:27.128571    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk" podStartSLOduration=2.128554385 podStartE2EDuration="2.128554385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:27.122259794 +0000 UTC m=+1434.714876593" watchObservedRunningTime="2026-11-02 15:20:27.128554385 +0000 UTC m=+1434.721171189"
Nov 02 15:20:28.019082 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.019040    4891 generic.go:356] "Generic (PLEG): container finished" podID="b18ef3b4-6a05-46ae-8a34-9c7535359b6b" containerID="e0bc94bada53baa514935114fde5dd5caae6ce25601fd16ca0004854c62107fe" exitCode=0
Nov 02 15:20:28.019601 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.019095    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk" event={"ID":"b18ef3b4-6a05-46ae-8a34-9c7535359b6b","Type":"ContainerDied","Data":"e0bc94bada53baa514935114fde5dd5caae6ce25601fd16ca0004854c62107fe"}
Nov 02 15:20:28.060346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.060289    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"]
Nov 02 15:20:28.110510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.109226    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a39269-j2glj"]
Nov 02 15:20:28.265016 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.264990    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:28.319370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.319136    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-xks-ca\") pod \"995af847-3a60-4464-910f-d8d66dbc3672\" (UID: \"995af847-3a60-4464-910f-d8d66dbc3672\") "
Nov 02 15:20:28.319635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.319612    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995af847-3a60-4464-910f-d8d66dbc3672-kube-api-access-tmjwx\" (UniqueName: \"kubernetes.io/projected/995af847-3a60-4464-910f-d8d66dbc3672-kube-api-access-tmjwx\") pod \"995af847-3a60-4464-910f-d8d66dbc3672\" (UID: \"995af847-3a60-4464-910f-d8d66dbc3672\") "
Nov 02 15:20:28.319720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.319690    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-openshift-ca\") pod \"995af847-3a60-4464-910f-d8d66dbc3672\" (UID: \"995af847-3a60-4464-910f-d8d66dbc3672\") "
Nov 02 15:20:28.320076 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.320040    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-openshift-ca" pod "995af847-3a60-4464-910f-d8d66dbc3672" (UID: "995af847-3a60-4464-910f-d8d66dbc3672"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:28.320434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.320413    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:28.321582 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.321560    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-xks-ca" pod "995af847-3a60-4464-910f-d8d66dbc3672" (UID: "995af847-3a60-4464-910f-d8d66dbc3672"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:20:28.321802 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.321782    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995af847-3a60-4464-910f-d8d66dbc3672-kube-api-access-tmjwx" pod "995af847-3a60-4464-910f-d8d66dbc3672" (UID: "995af847-3a60-4464-910f-d8d66dbc3672"). InnerVolumeSpecName "kube-api-access-tmjwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:28.421845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.421809    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmjwx\" (UniqueName: \"kubernetes.io/projected/995af847-3a60-4464-910f-d8d66dbc3672-kube-api-access-tmjwx\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:28.421845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.421843    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/995af847-3a60-4464-910f-d8d66dbc3672-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:28.525501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:28.525463    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af7f96e3-d48c-4ee2-a6d9-81677ddf708a" path="/var/lib/kubelet/pods/af7f96e3-d48c-4ee2-a6d9-81677ddf708a/volumes"
Nov 02 15:20:29.029661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.029620    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"
Nov 02 15:20:29.029661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.029654    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95" event={"ID":"995af847-3a60-4464-910f-d8d66dbc3672","Type":"ContainerDied","Data":"4283e80496e5466acf30074b9532b6872468339ba6f6726e33661f92a0dff184"}
Nov 02 15:20:29.030216 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.029700    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4283e80496e5466acf30074b9532b6872468339ba6f6726e33661f92a0dff184"
Nov 02 15:20:29.222656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.222628    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:29.233181 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.233158    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-xks-ca\") pod \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\" (UID: \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\") "
Nov 02 15:20:29.233285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.233190    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-openshift-ca\") pod \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\" (UID: \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\") "
Nov 02 15:20:29.233285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.233219    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-kube-api-access-jwwcw\" (UniqueName: \"kubernetes.io/projected/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-kube-api-access-jwwcw\") pod \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\" (UID: \"b18ef3b4-6a05-46ae-8a34-9c7535359b6b\") "
Nov 02 15:20:29.233625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.233595    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-openshift-ca" pod "b18ef3b4-6a05-46ae-8a34-9c7535359b6b" (UID: "b18ef3b4-6a05-46ae-8a34-9c7535359b6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:29.233740 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.233719    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:29.235368 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.235348    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-kube-api-access-jwwcw" pod "b18ef3b4-6a05-46ae-8a34-9c7535359b6b" (UID: "b18ef3b4-6a05-46ae-8a34-9c7535359b6b"). InnerVolumeSpecName "kube-api-access-jwwcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:29.235622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.235607    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-xks-ca" pod "b18ef3b4-6a05-46ae-8a34-9c7535359b6b" (UID: "b18ef3b4-6a05-46ae-8a34-9c7535359b6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:20:29.334818 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.334736    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:29.334818 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:29.334766    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwwcw\" (UniqueName: \"kubernetes.io/projected/b18ef3b4-6a05-46ae-8a34-9c7535359b6b-kube-api-access-jwwcw\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:30.039820 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:30.039779    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk" event={"ID":"b18ef3b4-6a05-46ae-8a34-9c7535359b6b","Type":"ContainerDied","Data":"4f69eea623f2019fd8c2796faa1d18d51f83989639f717e0d8a0f1abd201eddc"}
Nov 02 15:20:30.040216 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:30.039869    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f69eea623f2019fd8c2796faa1d18d51f83989639f717e0d8a0f1abd201eddc"
Nov 02 15:20:30.040216 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:30.039801    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"
Nov 02 15:20:31.805218 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.805181    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"]
Nov 02 15:20:31.806072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.806050    4891 cpu_manager.go:414] "containerMap: removing container" podUID="b18ef3b4-6a05-46ae-8a34-9c7535359b6b" containerName="revoke-keys"
Nov 02 15:20:31.806072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.806072    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18ef3b4-6a05-46ae-8a34-9c7535359b6b" containerName="revoke-keys"
Nov 02 15:20:31.806254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.806144    4891 cpu_manager.go:414] "containerMap: removing container" podUID="995af847-3a60-4464-910f-d8d66dbc3672" containerName="revoke-keys"
Nov 02 15:20:31.806254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.806153    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995af847-3a60-4464-910f-d8d66dbc3672" containerName="revoke-keys"
Nov 02 15:20:31.806254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.806165    4891 cpu_manager.go:414] "containerMap: removing container" podUID="64e14513-d6ca-4e5d-af48-fdfad82d8d45" containerName="revoke-keys"
Nov 02 15:20:31.806254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.806172    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e14513-d6ca-4e5d-af48-fdfad82d8d45" containerName="revoke-keys"
Nov 02 15:20:31.806478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.806390    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="995af847-3a60-4464-910f-d8d66dbc3672" containerName="revoke-keys"
Nov 02 15:20:31.806478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.806409    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e14513-d6ca-4e5d-af48-fdfad82d8d45" containerName="revoke-keys"
Nov 02 15:20:31.806478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.806424    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18ef3b4-6a05-46ae-8a34-9c7535359b6b" containerName="revoke-keys"
Nov 02 15:20:31.810304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.810284    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:31.865778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.865733    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:20:31.870518 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.870492    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dknt\" (UniqueName: \"kubernetes.io/projected/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-kube-api-access-9dknt\") pod \"maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8\" (UID: \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:31.870798 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.870772    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8\" (UID: \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:31.870888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.870874    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8\" (UID: \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:31.871617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.871596    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"]
Nov 02 15:20:31.912402 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.912373    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:20:31.912402 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.912373    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:20:31.915005 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.914981    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:20:31.971865 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.971827    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8\" (UID: \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:31.972043 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.971911    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8\" (UID: \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:31.972126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.972091    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dknt\" (UniqueName: \"kubernetes.io/projected/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-kube-api-access-9dknt\") pod \"maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8\" (UID: \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:31.972636 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.972614    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8\" (UID: \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:31.974855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:31.974831    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8\" (UID: \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:32.033102 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:32.033062    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dknt\" (UniqueName: \"kubernetes.io/projected/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-kube-api-access-9dknt\") pod \"maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8\" (UID: \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:32.135152 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:32.135064    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:32.593468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:32.593437    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"]
Nov 02 15:20:33.068809 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:33.068771    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8" event={"ID":"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6","Type":"ContainerStarted","Data":"b60782196ad61539c1113d20cf729c01fdf598386c1401bf45c04082cc0e5e66"}
Nov 02 15:20:33.069265 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:33.068827    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8" event={"ID":"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6","Type":"ContainerStarted","Data":"a663f38c54cd1a7014f78f16d2646af237c31a87b5313ba9ad01971210ff38ee"}
Nov 02 15:20:33.188014 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:33.187917    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8" podStartSLOduration=2.18790477 podStartE2EDuration="2.18790477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:33.183751902 +0000 UTC m=+1440.776368715" watchObservedRunningTime="2026-11-02 15:20:33.18790477 +0000 UTC m=+1440.780521570"
Nov 02 15:20:34.078748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:34.078712    4891 generic.go:356] "Generic (PLEG): container finished" podID="5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6" containerID="b60782196ad61539c1113d20cf729c01fdf598386c1401bf45c04082cc0e5e66" exitCode=0
Nov 02 15:20:34.078748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:34.078741    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8" event={"ID":"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6","Type":"ContainerDied","Data":"b60782196ad61539c1113d20cf729c01fdf598386c1401bf45c04082cc0e5e66"}
Nov 02 15:20:35.268911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:35.268882    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:35.348613 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:35.348518    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-openshift-ca\") pod \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\" (UID: \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\") "
Nov 02 15:20:35.348794 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:35.348623    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-xks-ca\") pod \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\" (UID: \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\") "
Nov 02 15:20:35.348794 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:35.348745    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-kube-api-access-9dknt\" (UniqueName: \"kubernetes.io/projected/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-kube-api-access-9dknt\") pod \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\" (UID: \"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6\") "
Nov 02 15:20:35.348980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:35.348949    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-openshift-ca" pod "5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6" (UID: "5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:35.349468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:35.349441    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:35.350859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:35.350831    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-xks-ca" pod "5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6" (UID: "5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:20:35.350982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:35.350962    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-kube-api-access-9dknt" pod "5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6" (UID: "5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6"). InnerVolumeSpecName "kube-api-access-9dknt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:35.451159 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:35.451124    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dknt\" (UniqueName: \"kubernetes.io/projected/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-kube-api-access-9dknt\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:35.451159 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:35.451158    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:36.097188 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:36.097156    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"
Nov 02 15:20:36.097385 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:36.097183    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8" event={"ID":"5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6","Type":"ContainerDied","Data":"a663f38c54cd1a7014f78f16d2646af237c31a87b5313ba9ad01971210ff38ee"}
Nov 02 15:20:36.097385 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:36.097232    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a663f38c54cd1a7014f78f16d2646af237c31a87b5313ba9ad01971210ff38ee"
Nov 02 15:20:44.001917 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:44.001877    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"]
Nov 02 15:20:44.275005 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:20:44.274962    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:20:49.002953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:49.002909    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv" podUID="bebebeb8-f4ed-47d6-a491-ceddc58fc949" containerName="payload-pre-processing" containerID="cri-o://96f00358ee187cdb5c308e5a23bc0b4f8abd9132bf103e8fe64c4d632e52738f" gracePeriod=25
Nov 02 15:20:49.221803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:49.220278    4891 generic.go:356] "Generic (PLEG): container finished" podID="bebebeb8-f4ed-47d6-a491-ceddc58fc949" containerID="96f00358ee187cdb5c308e5a23bc0b4f8abd9132bf103e8fe64c4d632e52738f" exitCode=0
Nov 02 15:20:49.221803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:49.220368    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv" event={"ID":"bebebeb8-f4ed-47d6-a491-ceddc58fc949","Type":"ContainerDied","Data":"96f00358ee187cdb5c308e5a23bc0b4f8abd9132bf103e8fe64c4d632e52738f"}
Nov 02 15:20:49.315812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:49.315772    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"
Nov 02 15:20:49.388219 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:49.388182    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bebebeb8-f4ed-47d6-a491-ceddc58fc949-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebebeb8-f4ed-47d6-a491-ceddc58fc949-plugins-config-volume\") pod \"bebebeb8-f4ed-47d6-a491-ceddc58fc949\" (UID: \"bebebeb8-f4ed-47d6-a491-ceddc58fc949\") "
Nov 02 15:20:49.388499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:49.388411    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebebeb8-f4ed-47d6-a491-ceddc58fc949-kube-api-access-wzplp\" (UniqueName: \"kubernetes.io/projected/bebebeb8-f4ed-47d6-a491-ceddc58fc949-kube-api-access-wzplp\") pod \"bebebeb8-f4ed-47d6-a491-ceddc58fc949\" (UID: \"bebebeb8-f4ed-47d6-a491-ceddc58fc949\") "
Nov 02 15:20:49.391172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:49.391135    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bebebeb8-f4ed-47d6-a491-ceddc58fc949-plugins-config-volume" pod "bebebeb8-f4ed-47d6-a491-ceddc58fc949" (UID: "bebebeb8-f4ed-47d6-a491-ceddc58fc949"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:49.391377 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:49.391246    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebebeb8-f4ed-47d6-a491-ceddc58fc949-kube-api-access-wzplp" pod "bebebeb8-f4ed-47d6-a491-ceddc58fc949" (UID: "bebebeb8-f4ed-47d6-a491-ceddc58fc949"). InnerVolumeSpecName "kube-api-access-wzplp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:49.490390 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:49.490354    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebebeb8-f4ed-47d6-a491-ceddc58fc949-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:49.490390 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:49.490384    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzplp\" (UniqueName: \"kubernetes.io/projected/bebebeb8-f4ed-47d6-a491-ceddc58fc949-kube-api-access-wzplp\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:20:50.230879 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:50.230844    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"
Nov 02 15:20:50.230879 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:50.230871    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv" event={"ID":"bebebeb8-f4ed-47d6-a491-ceddc58fc949","Type":"ContainerDied","Data":"d1a5bd5af261da2474c45f7916eac85ba0a4b55ea0a72e8668596520a196c93f"}
Nov 02 15:20:50.231635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:50.230947    4891 scope.go:122] "RemoveContainer" containerID="96f00358ee187cdb5c308e5a23bc0b4f8abd9132bf103e8fe64c4d632e52738f"
Nov 02 15:20:50.398531 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:50.398499    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"]
Nov 02 15:20:50.448145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:50.448110    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ccdeb27-7b96689bd7-n76hv"]
Nov 02 15:20:50.525959 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:50.525930    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bebebeb8-f4ed-47d6-a491-ceddc58fc949" path="/var/lib/kubelet/pods/bebebeb8-f4ed-47d6-a491-ceddc58fc949/volumes"
Nov 02 15:20:56.053903 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:56.053862    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"]
Nov 02 15:20:56.059522 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:56.059496    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b65040ec-jvlvb"]
Nov 02 15:20:56.525524 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:20:56.525494    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3197e5-9b98-49e0-8c06-32bfd41a7fbe" path="/var/lib/kubelet/pods/0d3197e5-9b98-49e0-8c06-32bfd41a7fbe/volumes"
Nov 02 15:21:07.051631 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:07.051592    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"]
Nov 02 15:21:07.100861 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:07.100831    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a9bba9-8xmmc"]
Nov 02 15:21:08.525586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:08.525547    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3329707-e6d4-45b2-a2ad-df6384d1c895" path="/var/lib/kubelet/pods/b3329707-e6d4-45b2-a2ad-df6384d1c895/volumes"
Nov 02 15:21:10.232703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:10.232672    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"]
Nov 02 15:21:10.281569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:10.281528    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad0b4b5e-ksvm9"]
Nov 02 15:21:10.581332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:10.559262    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e14513-d6ca-4e5d-af48-fdfad82d8d45" path="/var/lib/kubelet/pods/64e14513-d6ca-4e5d-af48-fdfad82d8d45/volumes"
Nov 02 15:21:11.831060 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:11.831009    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"]
Nov 02 15:21:11.964984 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:11.964951    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c82fc808-l59pk"]
Nov 02 15:21:12.966353 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:12.955853    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b18ef3b4-6a05-46ae-8a34-9c7535359b6b" path="/var/lib/kubelet/pods/b18ef3b4-6a05-46ae-8a34-9c7535359b6b/volumes"
Nov 02 15:21:13.123465 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:13.123402    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:13.429331 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:13.428437    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:13.459331 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:13.458394    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:13.482338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:13.474397    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 15:21:13.511335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:13.510459    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:13.530377 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:13.512490    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:13.814653 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:13.813431    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:13.819335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:13.817391    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:14.028645 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:14.028603    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Nov 02 15:21:14.470335 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:21:14.467503    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:21:20.989336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:20.979802    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"]
Nov 02 15:21:21.060632 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:21.060595    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ccdeb27-zcwk8"]
Nov 02 15:21:22.525422 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:22.525388    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6" path="/var/lib/kubelet/pods/5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6/volumes"
Nov 02 15:21:26.082761 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:26.082718    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"]
Nov 02 15:21:26.130905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:26.130868    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8c556b-ph47s"]
Nov 02 15:21:26.529622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:26.529590    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094d6dba-849b-4faa-b1f7-8725979638d1" path="/var/lib/kubelet/pods/094d6dba-849b-4faa-b1f7-8725979638d1/volumes"
Nov 02 15:21:33.051485 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.051451    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:21:33.053025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.053006    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:21:33.053169 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.053062    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:21:33.054035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.054015    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:21:33.068545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.068524    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:21:33.068704 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.068677    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:21:33.731382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.731343    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"]
Nov 02 15:21:33.732021 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.732004    4891 cpu_manager.go:414] "containerMap: removing container" podUID="bebebeb8-f4ed-47d6-a491-ceddc58fc949" containerName="payload-pre-processing"
Nov 02 15:21:33.732081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.732022    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bebebeb8-f4ed-47d6-a491-ceddc58fc949" containerName="payload-pre-processing"
Nov 02 15:21:33.732081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.732060    4891 cpu_manager.go:414] "containerMap: removing container" podUID="5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6" containerName="revoke-keys"
Nov 02 15:21:33.732081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.732065    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6" containerName="revoke-keys"
Nov 02 15:21:33.732196 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.732185    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="bebebeb8-f4ed-47d6-a491-ceddc58fc949" containerName="payload-pre-processing"
Nov 02 15:21:33.732235 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.732205    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f30a2fe-cb1f-4fc8-bd4f-edc2dc35e6f6" containerName="revoke-keys"
Nov 02 15:21:33.736185 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.736165    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:33.790871 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.790840    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 15:21:33.791050 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.791008    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 15:21:33.791050 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.791017    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 15:21:33.840233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.840203    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2kqpp\""
Nov 02 15:21:33.848346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.848299    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"]
Nov 02 15:21:33.902913 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.902878    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:33.903081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.902930    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:33.903125 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.903080    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7137ff72-f89d-46c8-95c7-34569d6cdd06-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:33.903160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.903137    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7skml\" (UniqueName: \"kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-kube-api-access-7skml\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:33.903334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.903295    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:33.903398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.903385    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:33.903444 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:33.903432    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.005128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.005034    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.005128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.005105    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.005424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.005208    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.005424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.005254    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.005424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.005329    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7137ff72-f89d-46c8-95c7-34569d6cdd06-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.005424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.005365    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7skml\" (UniqueName: \"kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-kube-api-access-7skml\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.005655 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.005534    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.006057 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.006030    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.007658 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.007624    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7137ff72-f89d-46c8-95c7-34569d6cdd06-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.007766 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.007705    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.007983 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.007963    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.008069 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.008045    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.044356 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.044303    4891 scope.go:122] "RemoveContainer" containerID="efc6014e59878cc5af346b60adf3f1a1bae84f6e2107b1c20a689921ac836835"
Nov 02 15:21:34.072364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.072300    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7skml\" (UniqueName: \"kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-kube-api-access-7skml\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.073100 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.073076    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hh82k\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.349537 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.349506    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:34.848860 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:21:34.848828    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7137ff72_f89d_46c8_95c7_34569d6cdd06.slice/crio-5d3f9edf5a303614831b6b81feab98844e2d1b7b3b12f9d2afa557bc7c73cc50 WatchSource:0}: Error finding container 5d3f9edf5a303614831b6b81feab98844e2d1b7b3b12f9d2afa557bc7c73cc50: Status 404 returned error can't find the container with id 5d3f9edf5a303614831b6b81feab98844e2d1b7b3b12f9d2afa557bc7c73cc50
Nov 02 15:21:34.850267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.850236    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"]
Nov 02 15:21:34.851017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:34.851001    4891 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:21:35.215348 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:35.215229    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k" event={"ID":"7137ff72-f89d-46c8-95c7-34569d6cdd06","Type":"ContainerStarted","Data":"5d3f9edf5a303614831b6b81feab98844e2d1b7b3b12f9d2afa557bc7c73cc50"}
Nov 02 15:21:38.580173 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:38.580127    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:38.580541 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:38.580203    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:39.062049 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:39.062018    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"]
Nov 02 15:21:39.112141 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:39.112103    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb05927-7hrq4"]
Nov 02 15:21:39.266832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:39.266793    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k" event={"ID":"7137ff72-f89d-46c8-95c7-34569d6cdd06","Type":"ContainerStarted","Data":"157022e3fdcc0bde7cf74786aeab873e77c732bbfa49804695442136f873a0f1"}
Nov 02 15:21:39.267115 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:39.267086    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:39.268587 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:39.268566    4891 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hh82k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:21:39.268681 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:39.268618    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k" podUID="7137ff72-f89d-46c8-95c7-34569d6cdd06" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:21:39.390735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:39.390633    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k" podStartSLOduration=2.66295851 podStartE2EDuration="6.390613291s" totalImagesPullingTime="3.727654781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:39.387045248 +0000 UTC m=+1506.979662051" watchObservedRunningTime="2026-11-02 15:21:39.390613291 +0000 UTC m=+1506.983230094"
Nov 02 15:21:40.307658 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:40.307614    4891 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hh82k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:21:40.308154 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:40.307688    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k" podUID="7137ff72-f89d-46c8-95c7-34569d6cdd06" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:21:40.525899 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:40.525862    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="956b7542-522f-4792-a4b8-8ec900796bab" path="/var/lib/kubelet/pods/956b7542-522f-4792-a4b8-8ec900796bab/volumes"
Nov 02 15:21:41.053697 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:41.053664    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"]
Nov 02 15:21:41.103986 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:41.103953    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06b85ca0-z9l47"]
Nov 02 15:21:42.526274 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:42.526238    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e" path="/var/lib/kubelet/pods/c57d1e3a-2dfb-4067-9b3b-e6d8b1bc6d7e/volumes"
Nov 02 15:21:43.308460 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:43.308424    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:21:44.275228 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:21:44.275186    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:21:55.185511 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:55.185473    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"]
Nov 02 15:21:55.234363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:55.231575    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32de075f-7zv58"]
Nov 02 15:21:56.364960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:56.364911    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:56.391199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:56.390407    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 15:21:56.447440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:56.447392    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:56.457335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:56.456181    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:56.516827 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:56.516546    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:56.517370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:56.517156    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:56.535343 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:56.533546    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 15:21:56.541759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:21:56.541725    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a" path="/var/lib/kubelet/pods/77fd0ace-8fc1-42d8-bbf4-2123ae0ff35a/volumes"
Nov 02 15:22:08.058716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:08.058609    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"]
Nov 02 15:22:08.072422 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:08.072384    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"]
Nov 02 15:22:08.078247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:08.078210    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8994365a-k4z28"]
Nov 02 15:22:08.108293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:08.108261    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d271c8a-xm727"]
Nov 02 15:22:08.524357 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:08.524327    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35212288-ebce-42b4-beab-807ca74c1562" path="/var/lib/kubelet/pods/35212288-ebce-42b4-beab-807ca74c1562/volumes"
Nov 02 15:22:08.524904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:08.524883    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc9aa5a-b8a5-4d64-9d78-0209730f77f5" path="/var/lib/kubelet/pods/ebc9aa5a-b8a5-4d64-9d78-0209730f77f5/volumes"
Nov 02 15:22:14.283238 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:22:14.283188    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:22:20.056792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:20.056752    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"]
Nov 02 15:22:20.106375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:20.106338    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca399e8f-xnp7f"]
Nov 02 15:22:20.540065 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:20.540025    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b" path="/var/lib/kubelet/pods/5d9a34f5-9b4e-4ea1-b3f6-ef123fc2275b/volumes"
Nov 02 15:22:20.880186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:20.880114    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"]
Nov 02 15:22:20.889262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:20.889233    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"
Nov 02 15:22:20.942738 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:20.942706    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"]
Nov 02 15:22:20.969426 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:20.969380    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6m6s\" (UniqueName: \"kubernetes.io/projected/d97704bf-338e-4982-85ab-770f8aafdb5c-kube-api-access-v6m6s\") pod \"payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2\" (UID: \"d97704bf-338e-4982-85ab-770f8aafdb5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"
Nov 02 15:22:20.969608 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:20.969477    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97704bf-338e-4982-85ab-770f8aafdb5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2\" (UID: \"d97704bf-338e-4982-85ab-770f8aafdb5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"
Nov 02 15:22:20.989047 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:20.989011    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-895ac8-dockercfg-qrw55\""
Nov 02 15:22:20.989233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:20.989079    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-895ac8\""
Nov 02 15:22:21.074356 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:21.074290    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6m6s\" (UniqueName: \"kubernetes.io/projected/d97704bf-338e-4982-85ab-770f8aafdb5c-kube-api-access-v6m6s\") pod \"payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2\" (UID: \"d97704bf-338e-4982-85ab-770f8aafdb5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"
Nov 02 15:22:21.075318 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:21.075282    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97704bf-338e-4982-85ab-770f8aafdb5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2\" (UID: \"d97704bf-338e-4982-85ab-770f8aafdb5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"
Nov 02 15:22:21.076467 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:21.076441    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97704bf-338e-4982-85ab-770f8aafdb5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2\" (UID: \"d97704bf-338e-4982-85ab-770f8aafdb5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"
Nov 02 15:22:21.145604 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:21.144198    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6m6s\" (UniqueName: \"kubernetes.io/projected/d97704bf-338e-4982-85ab-770f8aafdb5c-kube-api-access-v6m6s\") pod \"payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2\" (UID: \"d97704bf-338e-4982-85ab-770f8aafdb5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"
Nov 02 15:22:21.205616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:21.205584    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"
Nov 02 15:22:21.625671 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:21.625638    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"]
Nov 02 15:22:21.636641 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:22:21.636607    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97704bf_338e_4982_85ab_770f8aafdb5c.slice/crio-5cdfa6fbd00f4d7a41fa04d9a8e84cebd0d49f4aa354ace369bad9911dd93cfb WatchSource:0}: Error finding container 5cdfa6fbd00f4d7a41fa04d9a8e84cebd0d49f4aa354ace369bad9911dd93cfb: Status 404 returned error can't find the container with id 5cdfa6fbd00f4d7a41fa04d9a8e84cebd0d49f4aa354ace369bad9911dd93cfb
Nov 02 15:22:21.883484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:21.883389    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2" event={"ID":"d97704bf-338e-4982-85ab-770f8aafdb5c","Type":"ContainerStarted","Data":"58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85"}
Nov 02 15:22:21.883484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:21.883446    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2" event={"ID":"d97704bf-338e-4982-85ab-770f8aafdb5c","Type":"ContainerStarted","Data":"5cdfa6fbd00f4d7a41fa04d9a8e84cebd0d49f4aa354ace369bad9911dd93cfb"}
Nov 02 15:22:21.883700 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:21.883520    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"
Nov 02 15:22:22.003383 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:22.003306    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2" podStartSLOduration=2.003286767 podStartE2EDuration="2.003286767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:22.003092836 +0000 UTC m=+1549.595709634" watchObservedRunningTime="2026-11-02 15:22:22.003286767 +0000 UTC m=+1549.595903567"
Nov 02 15:22:25.051872 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:25.051832    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"]
Nov 02 15:22:25.101040 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:25.101009    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ae12f8-bn8c8"]
Nov 02 15:22:26.051751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:26.051715    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"]
Nov 02 15:22:26.103372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:26.103337    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0621a2db-m2w6p"]
Nov 02 15:22:26.524800 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:26.524769    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892a2e8d-6377-4b85-bc3e-198b97e39198" path="/var/lib/kubelet/pods/892a2e8d-6377-4b85-bc3e-198b97e39198/volumes"
Nov 02 15:22:26.525349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:26.525328    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f278d76b-0fae-4736-8073-9396d0228f10" path="/var/lib/kubelet/pods/f278d76b-0fae-4736-8073-9396d0228f10/volumes"
Nov 02 15:22:29.053118 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:29.053079    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"]
Nov 02 15:22:29.102709 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:29.102673    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b490704d-sv9xw"]
Nov 02 15:22:29.918128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:29.918091    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"]
Nov 02 15:22:29.924351 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:29.924290    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"
Nov 02 15:22:30.036781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.030018    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-b87e90\""
Nov 02 15:22:30.036781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.030869    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-b87e90-dockercfg-cvqgn\""
Nov 02 15:22:30.063340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.052245    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"]
Nov 02 15:22:30.077386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.076668    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"]
Nov 02 15:22:30.085409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.083693    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"
Nov 02 15:22:30.112648 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.112605    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8\" (UID: \"addd08e1-96c5-4e16-9d9f-9eb18b1a643c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"
Nov 02 15:22:30.112894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.112876    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnql6\" (UniqueName: \"kubernetes.io/projected/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-kube-api-access-rnql6\") pod \"payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8\" (UID: \"addd08e1-96c5-4e16-9d9f-9eb18b1a643c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"
Nov 02 15:22:30.137966 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.137930    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"]
Nov 02 15:22:30.214923 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.214298    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkhz5\" (UniqueName: \"kubernetes.io/projected/16b6843d-d0bb-474e-94c9-e3319e14b84d-kube-api-access-kkhz5\") pod \"payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5\" (UID: \"16b6843d-d0bb-474e-94c9-e3319e14b84d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"
Nov 02 15:22:30.214923 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.214466    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8\" (UID: \"addd08e1-96c5-4e16-9d9f-9eb18b1a643c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"
Nov 02 15:22:30.214923 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.214591    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b6843d-d0bb-474e-94c9-e3319e14b84d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5\" (UID: \"16b6843d-d0bb-474e-94c9-e3319e14b84d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"
Nov 02 15:22:30.214923 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.214773    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnql6\" (UniqueName: \"kubernetes.io/projected/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-kube-api-access-rnql6\") pod \"payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8\" (UID: \"addd08e1-96c5-4e16-9d9f-9eb18b1a643c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"
Nov 02 15:22:30.216078 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.216050    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8\" (UID: \"addd08e1-96c5-4e16-9d9f-9eb18b1a643c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"
Nov 02 15:22:30.276132 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.276100    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnql6\" (UniqueName: \"kubernetes.io/projected/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-kube-api-access-rnql6\") pod \"payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8\" (UID: \"addd08e1-96c5-4e16-9d9f-9eb18b1a643c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"
Nov 02 15:22:30.316143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.316102    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhz5\" (UniqueName: \"kubernetes.io/projected/16b6843d-d0bb-474e-94c9-e3319e14b84d-kube-api-access-kkhz5\") pod \"payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5\" (UID: \"16b6843d-d0bb-474e-94c9-e3319e14b84d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"
Nov 02 15:22:30.316370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.316330    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b6843d-d0bb-474e-94c9-e3319e14b84d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5\" (UID: \"16b6843d-d0bb-474e-94c9-e3319e14b84d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"
Nov 02 15:22:30.317408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.317381    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b6843d-d0bb-474e-94c9-e3319e14b84d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5\" (UID: \"16b6843d-d0bb-474e-94c9-e3319e14b84d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"
Nov 02 15:22:30.385359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.379324    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhz5\" (UniqueName: \"kubernetes.io/projected/16b6843d-d0bb-474e-94c9-e3319e14b84d-kube-api-access-kkhz5\") pod \"payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5\" (UID: \"16b6843d-d0bb-474e-94c9-e3319e14b84d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"
Nov 02 15:22:30.415198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.415149    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"
Nov 02 15:22:30.564334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.544652    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"
Nov 02 15:22:30.586450 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.567296    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02eeda9c-0ffc-495d-af44-76928db6e20d" path="/var/lib/kubelet/pods/02eeda9c-0ffc-495d-af44-76928db6e20d/volumes"
Nov 02 15:22:30.874152 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:22:30.874114    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b6843d_d0bb_474e_94c9_e3319e14b84d.slice/crio-e315b33310a997e3489e75ede1ddcb80c866b442c387d8c3583399b0923d1621 WatchSource:0}: Error finding container e315b33310a997e3489e75ede1ddcb80c866b442c387d8c3583399b0923d1621: Status 404 returned error can't find the container with id e315b33310a997e3489e75ede1ddcb80c866b442c387d8c3583399b0923d1621
Nov 02 15:22:30.874843 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.874817    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"]
Nov 02 15:22:30.981597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.981559    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5" event={"ID":"16b6843d-d0bb-474e-94c9-e3319e14b84d","Type":"ContainerStarted","Data":"2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17"}
Nov 02 15:22:30.981777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.981628    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5" event={"ID":"16b6843d-d0bb-474e-94c9-e3319e14b84d","Type":"ContainerStarted","Data":"e315b33310a997e3489e75ede1ddcb80c866b442c387d8c3583399b0923d1621"}
Nov 02 15:22:30.981777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:30.981659    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"
Nov 02 15:22:31.013986 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:22:31.013960    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaddd08e1_96c5_4e16_9d9f_9eb18b1a643c.slice/crio-e272f96612689d7afa4d984b2030732008c236c8cf270baf75be63146e132ceb WatchSource:0}: Error finding container e272f96612689d7afa4d984b2030732008c236c8cf270baf75be63146e132ceb: Status 404 returned error can't find the container with id e272f96612689d7afa4d984b2030732008c236c8cf270baf75be63146e132ceb
Nov 02 15:22:31.014870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:31.014845    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"]
Nov 02 15:22:31.103268 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:31.103217    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5" podStartSLOduration=1.1032045209999999 podStartE2EDuration="1.103204521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:31.098906305 +0000 UTC m=+1558.691523103" watchObservedRunningTime="2026-11-02 15:22:31.103204521 +0000 UTC m=+1558.695821320"
Nov 02 15:22:31.994733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:31.994683    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8" event={"ID":"addd08e1-96c5-4e16-9d9f-9eb18b1a643c","Type":"ContainerStarted","Data":"c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83"}
Nov 02 15:22:31.994939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:31.994754    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8" event={"ID":"addd08e1-96c5-4e16-9d9f-9eb18b1a643c","Type":"ContainerStarted","Data":"e272f96612689d7afa4d984b2030732008c236c8cf270baf75be63146e132ceb"}
Nov 02 15:22:31.994939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:31.994790    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"
Nov 02 15:22:32.112181 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:32.112127    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8" podStartSLOduration=3.112111199 podStartE2EDuration="3.112111199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:32.109302555 +0000 UTC m=+1559.701919354" watchObservedRunningTime="2026-11-02 15:22:32.112111199 +0000 UTC m=+1559.704728010"
Nov 02 15:22:32.896265 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:32.896234    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"
Nov 02 15:22:34.090458 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:34.090426    4891 scope.go:122] "RemoveContainer" containerID="38e640b1ea86eb4e70062a35f95ded9adc98f1ad9d8ac0ee095da5c02a11791a"
Nov 02 15:22:34.102698 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:34.102677    4891 scope.go:122] "RemoveContainer" containerID="a92d2eeb613201bdab589fdf9a5c8d5ed1d7f3a121820b9abfb06b0f8db5a003"
Nov 02 15:22:34.113352 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:34.113287    4891 scope.go:122] "RemoveContainer" containerID="f490cddd31b18874ae406beec6a6ba512797fb044dad97bcc3ddb870d915aa45"
Nov 02 15:22:37.059394 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:37.059362    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"]
Nov 02 15:22:37.064858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:37.064828    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b59314cf-qhdts"]
Nov 02 15:22:38.525494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:38.525449    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4698bf0f-670a-47f6-aa26-670fc0f151a8" path="/var/lib/kubelet/pods/4698bf0f-670a-47f6-aa26-670fc0f151a8/volumes"
Nov 02 15:22:40.053570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:40.053531    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"]
Nov 02 15:22:40.101484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:40.101450    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40374a74-wrnx8"]
Nov 02 15:22:40.526286 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:40.526256    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ef8e0fb-be13-408f-b2f7-69c70e569e0d" path="/var/lib/kubelet/pods/2ef8e0fb-be13-408f-b2f7-69c70e569e0d/volumes"
Nov 02 15:22:41.999113 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:41.999082    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"
Nov 02 15:22:43.009851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:43.009777    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"
Nov 02 15:22:43.053344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:43.053293    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"]
Nov 02 15:22:43.101629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:43.101592    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d76dd5-2k68z"]
Nov 02 15:22:44.275099 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:22:44.275055    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:22:44.523485 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:44.523450    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d211982c-f8e5-411d-9e60-ef9f6fe95bb2" path="/var/lib/kubelet/pods/d211982c-f8e5-411d-9e60-ef9f6fe95bb2/volumes"
Nov 02 15:22:48.055223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:48.055182    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"]
Nov 02 15:22:48.102283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:48.102248    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52fe2649-gbd2v"]
Nov 02 15:22:48.524436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:48.524400    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f838ec-13e3-4c0e-befa-d432b8294837" path="/var/lib/kubelet/pods/39f838ec-13e3-4c0e-befa-d432b8294837/volumes"
Nov 02 15:22:52.172957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:52.172921    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"]
Nov 02 15:22:52.173487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:52.173142    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" podUID="625e325c-1d99-4f2d-992a-e9abd72792e3" containerName="istio-proxy" containerID="cri-o://4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f" gracePeriod=30
Nov 02 15:22:53.050790 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:53.050748    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"]
Nov 02 15:22:53.100295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:53.100260    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f67bfb-hccq7"]
Nov 02 15:22:54.526545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:54.526510    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd" path="/var/lib/kubelet/pods/9ffe0cb7-14d1-4b00-bb3b-f81ba823e8cd/volumes"
Nov 02 15:22:57.672526 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.672493    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:22:57.724523 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.724484    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-podinfo\") pod \"625e325c-1d99-4f2d-992a-e9abd72792e3\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") "
Nov 02 15:22:57.724731 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.724625    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-envoy\") pod \"625e325c-1d99-4f2d-992a-e9abd72792e3\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") "
Nov 02 15:22:57.724731 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.724668    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-data\") pod \"625e325c-1d99-4f2d-992a-e9abd72792e3\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") "
Nov 02 15:22:57.724834 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.724748    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625e325c-1d99-4f2d-992a-e9abd72792e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625e325c-1d99-4f2d-992a-e9abd72792e3-istiod-ca-cert\") pod \"625e325c-1d99-4f2d-992a-e9abd72792e3\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") "
Nov 02 15:22:57.724883 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.724844    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-certs\") pod \"625e325c-1d99-4f2d-992a-e9abd72792e3\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") "
Nov 02 15:22:57.724934 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.724889    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-credential-socket\") pod \"625e325c-1d99-4f2d-992a-e9abd72792e3\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") "
Nov 02 15:22:57.724988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.724941    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-token\" (UniqueName: \"kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-token\") pod \"625e325c-1d99-4f2d-992a-e9abd72792e3\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") "
Nov 02 15:22:57.725035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.724991    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-socket\") pod \"625e325c-1d99-4f2d-992a-e9abd72792e3\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") "
Nov 02 15:22:57.725086 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.725050    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-kube-api-access-wt24c\" (UniqueName: \"kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-kube-api-access-wt24c\") pod \"625e325c-1d99-4f2d-992a-e9abd72792e3\" (UID: \"625e325c-1d99-4f2d-992a-e9abd72792e3\") "
Nov 02 15:22:57.725620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.725593    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625e325c-1d99-4f2d-992a-e9abd72792e3-istiod-ca-cert" pod "625e325c-1d99-4f2d-992a-e9abd72792e3" (UID: "625e325c-1d99-4f2d-992a-e9abd72792e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:57.726092 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.725955    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-credential-socket" pod "625e325c-1d99-4f2d-992a-e9abd72792e3" (UID: "625e325c-1d99-4f2d-992a-e9abd72792e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:57.726238 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.726151    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-certs" pod "625e325c-1d99-4f2d-992a-e9abd72792e3" (UID: "625e325c-1d99-4f2d-992a-e9abd72792e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:57.726610 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.726567    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-socket" pod "625e325c-1d99-4f2d-992a-e9abd72792e3" (UID: "625e325c-1d99-4f2d-992a-e9abd72792e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:57.726610 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.726586    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-data" pod "625e325c-1d99-4f2d-992a-e9abd72792e3" (UID: "625e325c-1d99-4f2d-992a-e9abd72792e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:57.727995 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.727969    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-envoy" pod "625e325c-1d99-4f2d-992a-e9abd72792e3" (UID: "625e325c-1d99-4f2d-992a-e9abd72792e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:57.728396 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.728373    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-podinfo" pod "625e325c-1d99-4f2d-992a-e9abd72792e3" (UID: "625e325c-1d99-4f2d-992a-e9abd72792e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:22:57.728520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.728504    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-kube-api-access-wt24c" pod "625e325c-1d99-4f2d-992a-e9abd72792e3" (UID: "625e325c-1d99-4f2d-992a-e9abd72792e3"). InnerVolumeSpecName "kube-api-access-wt24c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:57.728752 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.728726    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-token" pod "625e325c-1d99-4f2d-992a-e9abd72792e3" (UID: "625e325c-1d99-4f2d-992a-e9abd72792e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:57.827673 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.827635    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:22:57.827673 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.827670    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:22:57.827673 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.827681    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:22:57.827969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.827691    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:22:57.827969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.827701    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt24c\" (UniqueName: \"kubernetes.io/projected/625e325c-1d99-4f2d-992a-e9abd72792e3-kube-api-access-wt24c\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:22:57.827969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.827710    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:22:57.827969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.827718    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:22:57.827969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.827728    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625e325c-1d99-4f2d-992a-e9abd72792e3-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:22:57.827969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:57.827736    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625e325c-1d99-4f2d-992a-e9abd72792e3-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:22:58.298324 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:58.298273    4891 generic.go:356] "Generic (PLEG): container finished" podID="625e325c-1d99-4f2d-992a-e9abd72792e3" containerID="4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f" exitCode=0
Nov 02 15:22:58.298540 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:58.298336    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"
Nov 02 15:22:58.298540 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:58.298363    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" event={"ID":"625e325c-1d99-4f2d-992a-e9abd72792e3","Type":"ContainerDied","Data":"4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f"}
Nov 02 15:22:58.298540 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:58.298431    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb" event={"ID":"625e325c-1d99-4f2d-992a-e9abd72792e3","Type":"ContainerDied","Data":"21733ab8a4b8f58f80938a95a1b2b3d10c2f338f6d2ecc96b921765332d6ab82"}
Nov 02 15:22:58.298540 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:58.298462    4891 scope.go:122] "RemoveContainer" containerID="4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f"
Nov 02 15:22:58.312214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:58.312192    4891 scope.go:122] "RemoveContainer" containerID="4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f"
Nov 02 15:22:58.312573 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:22:58.312549    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f\": container with ID starting with 4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f not found: ID does not exist" containerID="4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f"
Nov 02 15:22:58.312678 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:58.312580    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f"} err="failed to get container status \"4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f\": rpc error: code = NotFound desc = could not find container \"4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f\": container with ID starting with 4060099a8b361576e5f292975c96c6b98b1a36269d39753d26b54f3551cb3a5f not found: ID does not exist"
Nov 02 15:22:58.473657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:58.473621    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"]
Nov 02 15:22:58.527142 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:58.527096    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-rvvqb"]
Nov 02 15:22:59.053249 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:59.053216    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"]
Nov 02 15:22:59.058057 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:22:59.058022    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1913046a-8sm5g"]
Nov 02 15:23:00.529534 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.529498    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625e325c-1d99-4f2d-992a-e9abd72792e3" path="/var/lib/kubelet/pods/625e325c-1d99-4f2d-992a-e9abd72792e3/volumes"
Nov 02 15:23:00.530198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.530177    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2384dc8-971f-4170-9779-a2db56e05436" path="/var/lib/kubelet/pods/c2384dc8-971f-4170-9779-a2db56e05436/volumes"
Nov 02 15:23:00.584218 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.584185    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"]
Nov 02 15:23:00.584885 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.584871    4891 cpu_manager.go:414] "containerMap: removing container" podUID="625e325c-1d99-4f2d-992a-e9abd72792e3" containerName="istio-proxy"
Nov 02 15:23:00.584941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.584888    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625e325c-1d99-4f2d-992a-e9abd72792e3" containerName="istio-proxy"
Nov 02 15:23:00.584996 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.584987    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="625e325c-1d99-4f2d-992a-e9abd72792e3" containerName="istio-proxy"
Nov 02 15:23:00.590030 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.590004    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:00.647291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.647259    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"]
Nov 02 15:23:00.672250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.672205    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nplf8\" (UniqueName: \"kubernetes.io/projected/c34dbb25-8f76-4001-b488-98f13f3f0ffc-kube-api-access-nplf8\") pod \"maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4\" (UID: \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:00.672523 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.672474    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4\" (UID: \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:00.672629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.672595    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4\" (UID: \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:00.690490 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.690461    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:23:00.690683 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.690541    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:23:00.690683 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.690634    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:23:00.692163 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.692115    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:23:00.773966 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.773930    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nplf8\" (UniqueName: \"kubernetes.io/projected/c34dbb25-8f76-4001-b488-98f13f3f0ffc-kube-api-access-nplf8\") pod \"maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4\" (UID: \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:00.774142 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.774061    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4\" (UID: \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:00.774142 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.774129    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4\" (UID: \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:00.774776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.774753    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4\" (UID: \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:00.776695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.776665    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4\" (UID: \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:00.835538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.835468    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nplf8\" (UniqueName: \"kubernetes.io/projected/c34dbb25-8f76-4001-b488-98f13f3f0ffc-kube-api-access-nplf8\") pod \"maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4\" (UID: \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:00.905045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:00.905002    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:01.290717 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:23:01.290685    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34dbb25_8f76_4001_b488_98f13f3f0ffc.slice/crio-8d0c12cddbd3cacd8c09d2589cdbf6493a2b747624d2ab1fdd364d43af6203dc WatchSource:0}: Error finding container 8d0c12cddbd3cacd8c09d2589cdbf6493a2b747624d2ab1fdd364d43af6203dc: Status 404 returned error can't find the container with id 8d0c12cddbd3cacd8c09d2589cdbf6493a2b747624d2ab1fdd364d43af6203dc
Nov 02 15:23:01.292377 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:01.292245    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"]
Nov 02 15:23:01.331046 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:01.331015    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4" event={"ID":"c34dbb25-8f76-4001-b488-98f13f3f0ffc","Type":"ContainerStarted","Data":"8d0c12cddbd3cacd8c09d2589cdbf6493a2b747624d2ab1fdd364d43af6203dc"}
Nov 02 15:23:02.068148 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:02.068107    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"]
Nov 02 15:23:02.116342 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:02.116283    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5ffb06-lh4bk"]
Nov 02 15:23:02.341464 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:02.341368    4891 generic.go:356] "Generic (PLEG): container finished" podID="c34dbb25-8f76-4001-b488-98f13f3f0ffc" containerID="695ad821b84b08580c481d1b4bd52ca2370a28a236d7ee6350f3473e1f26a070" exitCode=0
Nov 02 15:23:02.341625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:02.341452    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4" event={"ID":"c34dbb25-8f76-4001-b488-98f13f3f0ffc","Type":"ContainerDied","Data":"695ad821b84b08580c481d1b4bd52ca2370a28a236d7ee6350f3473e1f26a070"}
Nov 02 15:23:02.563978 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:02.563941    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebb1f524-c2db-4ae8-a4de-6a52efd31d86" path="/var/lib/kubelet/pods/ebb1f524-c2db-4ae8-a4de-6a52efd31d86/volumes"
Nov 02 15:23:03.069435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.069363    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"]
Nov 02 15:23:03.075410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.075378    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"]
Nov 02 15:23:03.118379 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.118339    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63193799-pt42x"]
Nov 02 15:23:03.124427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.124395    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9018a125-mhmd7"]
Nov 02 15:23:03.542525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.542499    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:03.625486 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.625384    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-openshift-ca\") pod \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\" (UID: \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\") "
Nov 02 15:23:03.625486 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.625439    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34dbb25-8f76-4001-b488-98f13f3f0ffc-kube-api-access-nplf8\" (UniqueName: \"kubernetes.io/projected/c34dbb25-8f76-4001-b488-98f13f3f0ffc-kube-api-access-nplf8\") pod \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\" (UID: \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\") "
Nov 02 15:23:03.625789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.625502    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-xks-ca\") pod \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\" (UID: \"c34dbb25-8f76-4001-b488-98f13f3f0ffc\") "
Nov 02 15:23:03.625789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.625753    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-openshift-ca" pod "c34dbb25-8f76-4001-b488-98f13f3f0ffc" (UID: "c34dbb25-8f76-4001-b488-98f13f3f0ffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:03.626392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.626363    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:23:03.627684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.627657    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-xks-ca" pod "c34dbb25-8f76-4001-b488-98f13f3f0ffc" (UID: "c34dbb25-8f76-4001-b488-98f13f3f0ffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:03.627793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.627751    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34dbb25-8f76-4001-b488-98f13f3f0ffc-kube-api-access-nplf8" pod "c34dbb25-8f76-4001-b488-98f13f3f0ffc" (UID: "c34dbb25-8f76-4001-b488-98f13f3f0ffc"). InnerVolumeSpecName "kube-api-access-nplf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:03.727911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.727876    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nplf8\" (UniqueName: \"kubernetes.io/projected/c34dbb25-8f76-4001-b488-98f13f3f0ffc-kube-api-access-nplf8\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:23:03.727911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:03.727910    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34dbb25-8f76-4001-b488-98f13f3f0ffc-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:23:04.362629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:04.362598    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"
Nov 02 15:23:04.363054 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:04.362632    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4" event={"ID":"c34dbb25-8f76-4001-b488-98f13f3f0ffc","Type":"ContainerDied","Data":"8d0c12cddbd3cacd8c09d2589cdbf6493a2b747624d2ab1fdd364d43af6203dc"}
Nov 02 15:23:04.363054 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:04.362686    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d0c12cddbd3cacd8c09d2589cdbf6493a2b747624d2ab1fdd364d43af6203dc"
Nov 02 15:23:04.524805 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:04.524765    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fff663b-d37d-4b6d-b5a1-2dd0c713421f" path="/var/lib/kubelet/pods/3fff663b-d37d-4b6d-b5a1-2dd0c713421f/volumes"
Nov 02 15:23:04.525560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:04.525538    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c329fb-a342-4969-9858-54636369dd86" path="/var/lib/kubelet/pods/d8c329fb-a342-4969-9858-54636369dd86/volumes"
Nov 02 15:23:06.370694 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.370656    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"]
Nov 02 15:23:06.371343 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.371329    4891 cpu_manager.go:414] "containerMap: removing container" podUID="c34dbb25-8f76-4001-b488-98f13f3f0ffc" containerName="revoke-keys"
Nov 02 15:23:06.371395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.371344    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34dbb25-8f76-4001-b488-98f13f3f0ffc" containerName="revoke-keys"
Nov 02 15:23:06.371482 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.371473    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34dbb25-8f76-4001-b488-98f13f3f0ffc" containerName="revoke-keys"
Nov 02 15:23:06.376401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.376382    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:06.436158 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.436123    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"]
Nov 02 15:23:06.476098 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.476061    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn\" (UID: \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:06.476276 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.476221    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh9ls\" (UniqueName: \"kubernetes.io/projected/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-kube-api-access-mh9ls\") pod \"maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn\" (UID: \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:06.476507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.476482    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn\" (UID: \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:06.476832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.476817    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:23:06.477123 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.477103    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:23:06.477198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.477164    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:23:06.478135 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.478118    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:23:06.577465 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.577422    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh9ls\" (UniqueName: \"kubernetes.io/projected/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-kube-api-access-mh9ls\") pod \"maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn\" (UID: \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:06.577682 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.577664    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn\" (UID: \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:06.577744 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.577727    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn\" (UID: \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:06.578379 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.578360    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn\" (UID: \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:06.580114 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.580090    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn\" (UID: \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:06.638265 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.638192    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh9ls\" (UniqueName: \"kubernetes.io/projected/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-kube-api-access-mh9ls\") pod \"maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn\" (UID: \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:06.693261 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:06.693229    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:07.083536 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:07.083500    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"]
Nov 02 15:23:07.085370 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:23:07.085337    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3831f2_0536_4e9c_ab0f_9c398e904fb8.slice/crio-048e89bcd77565205de82f4805f8b3d54a72453b0667a4a6679621cde36b1274 WatchSource:0}: Error finding container 048e89bcd77565205de82f4805f8b3d54a72453b0667a4a6679621cde36b1274: Status 404 returned error can't find the container with id 048e89bcd77565205de82f4805f8b3d54a72453b0667a4a6679621cde36b1274
Nov 02 15:23:07.395563 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:07.395527    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn" event={"ID":"5b3831f2-0536-4e9c-ab0f-9c398e904fb8","Type":"ContainerStarted","Data":"ec623f024651514ad9e19c10156b3cac55d15f9f75dd1a964261f986b2af9062"}
Nov 02 15:23:07.395945 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:07.395592    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn" event={"ID":"5b3831f2-0536-4e9c-ab0f-9c398e904fb8","Type":"ContainerStarted","Data":"048e89bcd77565205de82f4805f8b3d54a72453b0667a4a6679621cde36b1274"}
Nov 02 15:23:07.516904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:07.516839    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn" podStartSLOduration=1.516824113 podStartE2EDuration="1.516824113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:07.511532351 +0000 UTC m=+1595.104149149" watchObservedRunningTime="2026-11-02 15:23:07.516824113 +0000 UTC m=+1595.109440937"
Nov 02 15:23:08.410446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:08.410365    4891 generic.go:356] "Generic (PLEG): container finished" podID="5b3831f2-0536-4e9c-ab0f-9c398e904fb8" containerID="ec623f024651514ad9e19c10156b3cac55d15f9f75dd1a964261f986b2af9062" exitCode=0
Nov 02 15:23:08.410835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:08.410623    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn" event={"ID":"5b3831f2-0536-4e9c-ab0f-9c398e904fb8","Type":"ContainerDied","Data":"ec623f024651514ad9e19c10156b3cac55d15f9f75dd1a964261f986b2af9062"}
Nov 02 15:23:09.058189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:09.058155    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"]
Nov 02 15:23:09.063101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:09.063073    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b47ac47-p7kvm"]
Nov 02 15:23:09.593587 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:09.593559    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:09.626105 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:09.626073    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-xks-ca\") pod \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\" (UID: \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\") "
Nov 02 15:23:09.626262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:09.626127    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-openshift-ca\") pod \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\" (UID: \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\") "
Nov 02 15:23:09.626262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:09.626231    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-kube-api-access-mh9ls\" (UniqueName: \"kubernetes.io/projected/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-kube-api-access-mh9ls\") pod \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\" (UID: \"5b3831f2-0536-4e9c-ab0f-9c398e904fb8\") "
Nov 02 15:23:09.626568 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:09.626538    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-openshift-ca" pod "5b3831f2-0536-4e9c-ab0f-9c398e904fb8" (UID: "5b3831f2-0536-4e9c-ab0f-9c398e904fb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:09.626948 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:09.626919    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:23:09.628349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:09.628326    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-xks-ca" pod "5b3831f2-0536-4e9c-ab0f-9c398e904fb8" (UID: "5b3831f2-0536-4e9c-ab0f-9c398e904fb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:09.628451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:09.628373    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-kube-api-access-mh9ls" pod "5b3831f2-0536-4e9c-ab0f-9c398e904fb8" (UID: "5b3831f2-0536-4e9c-ab0f-9c398e904fb8"). InnerVolumeSpecName "kube-api-access-mh9ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:09.739369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:09.736725    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:23:09.739369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:09.736775    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh9ls\" (UniqueName: \"kubernetes.io/projected/5b3831f2-0536-4e9c-ab0f-9c398e904fb8-kube-api-access-mh9ls\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:23:10.429718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:10.429676    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn" event={"ID":"5b3831f2-0536-4e9c-ab0f-9c398e904fb8","Type":"ContainerDied","Data":"048e89bcd77565205de82f4805f8b3d54a72453b0667a4a6679621cde36b1274"}
Nov 02 15:23:10.429902 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:10.429757    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="048e89bcd77565205de82f4805f8b3d54a72453b0667a4a6679621cde36b1274"
Nov 02 15:23:10.429902 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:10.429702    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"
Nov 02 15:23:10.527464 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:10.527432    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3badbb25-fded-43dd-aae7-243e2e0311b0" path="/var/lib/kubelet/pods/3badbb25-fded-43dd-aae7-243e2e0311b0/volumes"
Nov 02 15:23:14.274749 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:23:14.274709    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:23:19.055846 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:19.055807    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"]
Nov 02 15:23:19.105907 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:19.105873    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"]
Nov 02 15:23:19.111282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:19.111255    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa8a91e-cpq28"]
Nov 02 15:23:19.115792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:19.115766    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fd60c5-8w5m2"]
Nov 02 15:23:20.524975 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:20.524938    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307f7aec-bc96-4e78-95db-134ebf285a8e" path="/var/lib/kubelet/pods/307f7aec-bc96-4e78-95db-134ebf285a8e/volumes"
Nov 02 15:23:20.525691 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:20.525669    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454dc6bb-de62-4247-9127-70bc5d5d0a1e" path="/var/lib/kubelet/pods/454dc6bb-de62-4247-9127-70bc5d5d0a1e/volumes"
Nov 02 15:23:30.060959 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:30.060920    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"]
Nov 02 15:23:30.110228 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:30.110194    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1414d6c4-dbwb4"]
Nov 02 15:23:30.525589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:30.525554    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ae8c6f-6385-4221-9f82-094ca468bec0" path="/var/lib/kubelet/pods/b4ae8c6f-6385-4221-9f82-094ca468bec0/volumes"
Nov 02 15:23:34.172464 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:34.172433    4891 scope.go:122] "RemoveContainer" containerID="1723035ec8d5d46c091856c957dd481f7aa3b956be1e55f3897c8a91e6c2cfb8"
Nov 02 15:23:34.183580 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:34.183556    4891 scope.go:122] "RemoveContainer" containerID="8bbb993535230a5039fbfe08f80d2f66c210a9b4da57bb5694ba09f3c94f7e33"
Nov 02 15:23:34.194265 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:34.194245    4891 scope.go:122] "RemoveContainer" containerID="7ecef08172e9f12eadd47646e5166d4956d36bfe49905aa4c60ed729ee36d07a"
Nov 02 15:23:34.203829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:34.203812    4891 scope.go:122] "RemoveContainer" containerID="69474b05e2a2f29c86d29c4e855af89c2f3f145cdcc53fe89d16515f70fa98a1"
Nov 02 15:23:34.214050 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:34.214029    4891 scope.go:122] "RemoveContainer" containerID="cc3337d62b6f07117967e5f108c9daa4bc7f2713eb53a96f19e21c3d77f9e436"
Nov 02 15:23:34.223867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:34.223845    4891 scope.go:122] "RemoveContainer" containerID="17d7b766599897906db4303849c8da1b69a5b579e0c6faaab2008512af66f800"
Nov 02 15:23:34.233344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:34.233303    4891 scope.go:122] "RemoveContainer" containerID="1961e527f9b37c5017cc2149ad74ee5bccb91b4997ebe765c9184c22f799ad1c"
Nov 02 15:23:34.263289 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:34.263260    4891 scope.go:122] "RemoveContainer" containerID="98620c90e704c462736bcb5ce16b7f90671fdd31bd7a23761377ffcf281f81ce"
Nov 02 15:23:34.288353 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:34.288288    4891 scope.go:122] "RemoveContainer" containerID="687da2b75e1df7344a210b9f28897a69edadad7abf5bd7566529d14d1f901e66"
Nov 02 15:23:34.299163 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:34.299141    4891 scope.go:122] "RemoveContainer" containerID="e05c47c73a5587a5e51b1b9e7257583c7c0e7727f6dc774c0a6c70c891e18e7b"
Nov 02 15:23:36.056743 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:36.056706    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"]
Nov 02 15:23:36.106446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:36.106408    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3cb7a2c-mcbww"]
Nov 02 15:23:36.573334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:36.570585    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91c5ffc-c65a-4b39-bae9-09769108d54e" path="/var/lib/kubelet/pods/b91c5ffc-c65a-4b39-bae9-09769108d54e/volumes"
Nov 02 15:23:43.054929 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:43.054891    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"]
Nov 02 15:23:43.102767 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:43.102732    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77aaa20-6v8gv"]
Nov 02 15:23:44.290176 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:23:44.290140    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:23:44.526450 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:44.526410    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d38b34cf-4981-4932-82f8-b3d2856a5c00" path="/var/lib/kubelet/pods/d38b34cf-4981-4932-82f8-b3d2856a5c00/volumes"
Nov 02 15:23:47.062529 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:47.062490    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"]
Nov 02 15:23:47.112192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:47.112159    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c4aa69-sgcfq"]
Nov 02 15:23:48.054418 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:48.054370    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"]
Nov 02 15:23:48.103855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:48.103817    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b6bf862-nnpqd"]
Nov 02 15:23:48.525174 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:48.525134    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a555e6ba-dc5d-45ac-89da-ed00a0205f65" path="/var/lib/kubelet/pods/a555e6ba-dc5d-45ac-89da-ed00a0205f65/volumes"
Nov 02 15:23:48.525643 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:48.525628    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b" path="/var/lib/kubelet/pods/ef40a5ee-b4e0-4a7a-bdeb-2f13de573d4b/volumes"
Nov 02 15:23:58.060810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:58.060778    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"]
Nov 02 15:23:58.110110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:58.110074    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8f1da9b-nvpfv"]
Nov 02 15:23:58.532343 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:23:58.532276    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff03e96-2b02-46cf-b634-2f7acca977b5" path="/var/lib/kubelet/pods/fff03e96-2b02-46cf-b634-2f7acca977b5/volumes"
Nov 02 15:24:01.039011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:01.038898    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"]
Nov 02 15:24:01.084807 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:01.084766    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cbda1cc-pwbv4"]
Nov 02 15:24:02.386682 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:02.386642    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"]
Nov 02 15:24:02.435776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:02.435736    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-748f2e23-bsjfn"]
Nov 02 15:24:02.524221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:02.524191    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3831f2-0536-4e9c-ab0f-9c398e904fb8" path="/var/lib/kubelet/pods/5b3831f2-0536-4e9c-ab0f-9c398e904fb8/volumes"
Nov 02 15:24:02.524772 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:02.524753    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34dbb25-8f76-4001-b488-98f13f3f0ffc" path="/var/lib/kubelet/pods/c34dbb25-8f76-4001-b488-98f13f3f0ffc/volumes"
Nov 02 15:24:14.436954 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:24:14.436752    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:24:24.098368 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.098332    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"]
Nov 02 15:24:24.098999 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.098980    4891 cpu_manager.go:414] "containerMap: removing container" podUID="5b3831f2-0536-4e9c-ab0f-9c398e904fb8" containerName="revoke-keys"
Nov 02 15:24:24.098999 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.098998    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3831f2-0536-4e9c-ab0f-9c398e904fb8" containerName="revoke-keys"
Nov 02 15:24:24.099199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.099185    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3831f2-0536-4e9c-ab0f-9c398e904fb8" containerName="revoke-keys"
Nov 02 15:24:24.103670 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.103632    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:24.165793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.165753    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"]
Nov 02 15:24:24.205595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.205549    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:24:24.205595 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.205574    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:24:24.205810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.205604    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:24:24.207465 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.207440    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:24:24.213145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.213117    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjnpk\" (UniqueName: \"kubernetes.io/projected/25b284eb-d840-4103-9ef8-7115157d7333-kube-api-access-qjnpk\") pod \"maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8\" (UID: \"25b284eb-d840-4103-9ef8-7115157d7333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:24.213436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.213415    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8\" (UID: \"25b284eb-d840-4103-9ef8-7115157d7333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:24.213636 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.213617    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8\" (UID: \"25b284eb-d840-4103-9ef8-7115157d7333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:24.315179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.315138    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjnpk\" (UniqueName: \"kubernetes.io/projected/25b284eb-d840-4103-9ef8-7115157d7333-kube-api-access-qjnpk\") pod \"maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8\" (UID: \"25b284eb-d840-4103-9ef8-7115157d7333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:24.315434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.315274    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8\" (UID: \"25b284eb-d840-4103-9ef8-7115157d7333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:24.315530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.315492    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8\" (UID: \"25b284eb-d840-4103-9ef8-7115157d7333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:24.316211 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.316185    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8\" (UID: \"25b284eb-d840-4103-9ef8-7115157d7333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:24.318245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.318219    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8\" (UID: \"25b284eb-d840-4103-9ef8-7115157d7333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:24.384467 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.384387    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjnpk\" (UniqueName: \"kubernetes.io/projected/25b284eb-d840-4103-9ef8-7115157d7333-kube-api-access-qjnpk\") pod \"maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8\" (UID: \"25b284eb-d840-4103-9ef8-7115157d7333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:24.429621 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.429586    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:24.856342 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:24.856291    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"]
Nov 02 15:24:24.856869 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:24:24.856835    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b284eb_d840_4103_9ef8_7115157d7333.slice/crio-7330a76d07b75f30bf707eec76e5b73829e31f9319dcc787128ea27aa3fe5c70 WatchSource:0}: Error finding container 7330a76d07b75f30bf707eec76e5b73829e31f9319dcc787128ea27aa3fe5c70: Status 404 returned error can't find the container with id 7330a76d07b75f30bf707eec76e5b73829e31f9319dcc787128ea27aa3fe5c70
Nov 02 15:24:25.087324 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.087266    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"]
Nov 02 15:24:25.094461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.094431    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:25.156368 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.156331    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"]
Nov 02 15:24:25.259603 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.259575    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16f0068-8ks7w\" (UID: \"830e624c-98fb-48c6-a9b1-e798013a7392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:25.259798 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.259775    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16f0068-8ks7w\" (UID: \"830e624c-98fb-48c6-a9b1-e798013a7392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:25.260002 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.259985    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2bq\" (UniqueName: \"kubernetes.io/projected/830e624c-98fb-48c6-a9b1-e798013a7392-kube-api-access-jd2bq\") pod \"maas-api-revoke-sub-d16f0068-8ks7w\" (UID: \"830e624c-98fb-48c6-a9b1-e798013a7392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:25.361712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.361614    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16f0068-8ks7w\" (UID: \"830e624c-98fb-48c6-a9b1-e798013a7392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:25.361889 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.361754    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16f0068-8ks7w\" (UID: \"830e624c-98fb-48c6-a9b1-e798013a7392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:25.361949 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.361938    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2bq\" (UniqueName: \"kubernetes.io/projected/830e624c-98fb-48c6-a9b1-e798013a7392-kube-api-access-jd2bq\") pod \"maas-api-revoke-sub-d16f0068-8ks7w\" (UID: \"830e624c-98fb-48c6-a9b1-e798013a7392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:25.362494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.362474    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16f0068-8ks7w\" (UID: \"830e624c-98fb-48c6-a9b1-e798013a7392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:25.363994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.363972    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16f0068-8ks7w\" (UID: \"830e624c-98fb-48c6-a9b1-e798013a7392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:25.422628 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.422595    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2bq\" (UniqueName: \"kubernetes.io/projected/830e624c-98fb-48c6-a9b1-e798013a7392-kube-api-access-jd2bq\") pod \"maas-api-revoke-sub-d16f0068-8ks7w\" (UID: \"830e624c-98fb-48c6-a9b1-e798013a7392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:25.526646 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.526602    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8" event={"ID":"25b284eb-d840-4103-9ef8-7115157d7333","Type":"ContainerStarted","Data":"a6258f31d5b8c44cef6544147a77c5a7f698a98d641160161fe786611fa704aa"}
Nov 02 15:24:25.526828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.526671    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8" event={"ID":"25b284eb-d840-4103-9ef8-7115157d7333","Type":"ContainerStarted","Data":"7330a76d07b75f30bf707eec76e5b73829e31f9319dcc787128ea27aa3fe5c70"}
Nov 02 15:24:25.658269 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.658114    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8" podStartSLOduration=1.658093781 podStartE2EDuration="1.658093781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:25.65491992 +0000 UTC m=+1673.247536719" watchObservedRunningTime="2026-11-02 15:24:25.658093781 +0000 UTC m=+1673.250710581"
Nov 02 15:24:25.712548 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:25.712511    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:26.157858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:26.157819    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"]
Nov 02 15:24:26.160519 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:24:26.160470    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830e624c_98fb_48c6_a9b1_e798013a7392.slice/crio-00aaeeef5b2f267cbe7327042941b593618196c3fdfe9e42903edb16155ca112 WatchSource:0}: Error finding container 00aaeeef5b2f267cbe7327042941b593618196c3fdfe9e42903edb16155ca112: Status 404 returned error can't find the container with id 00aaeeef5b2f267cbe7327042941b593618196c3fdfe9e42903edb16155ca112
Nov 02 15:24:26.604035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:26.604002    4891 generic.go:356] "Generic (PLEG): container finished" podID="25b284eb-d840-4103-9ef8-7115157d7333" containerID="a6258f31d5b8c44cef6544147a77c5a7f698a98d641160161fe786611fa704aa" exitCode=0
Nov 02 15:24:26.605519 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:26.604250    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8" event={"ID":"25b284eb-d840-4103-9ef8-7115157d7333","Type":"ContainerDied","Data":"a6258f31d5b8c44cef6544147a77c5a7f698a98d641160161fe786611fa704aa"}
Nov 02 15:24:26.606562 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:26.606528    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w" event={"ID":"830e624c-98fb-48c6-a9b1-e798013a7392","Type":"ContainerStarted","Data":"00aaeeef5b2f267cbe7327042941b593618196c3fdfe9e42903edb16155ca112"}
Nov 02 15:24:27.016104 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.016010    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"]
Nov 02 15:24:27.020913 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.020888    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"
Nov 02 15:24:27.074362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.074275    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-de9c4ba4\""
Nov 02 15:24:27.082301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.078067    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-de9c4ba4-dockercfg-5zl2w\""
Nov 02 15:24:27.084815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.084781    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"]
Nov 02 15:24:27.107365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.107279    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkgnl\" (UniqueName: \"kubernetes.io/projected/7390b194-bcc2-4020-98c4-73cfb8d07480-kube-api-access-hkgnl\") pod \"payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r\" (UID: \"7390b194-bcc2-4020-98c4-73cfb8d07480\") " pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"
Nov 02 15:24:27.107583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.107483    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7390b194-bcc2-4020-98c4-73cfb8d07480-plugins-config-volume\") pod \"payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r\" (UID: \"7390b194-bcc2-4020-98c4-73cfb8d07480\") " pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"
Nov 02 15:24:27.167725 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.167694    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"]
Nov 02 15:24:27.172794 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.172757    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"
Nov 02 15:24:27.220386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.209719    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgnl\" (UniqueName: \"kubernetes.io/projected/7390b194-bcc2-4020-98c4-73cfb8d07480-kube-api-access-hkgnl\") pod \"payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r\" (UID: \"7390b194-bcc2-4020-98c4-73cfb8d07480\") " pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"
Nov 02 15:24:27.220386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.210184    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7\" (UID: \"8ecc4598-19ce-463b-a3de-5ee6ba41a08f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"
Nov 02 15:24:27.220386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.210232    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcfhh\" (UniqueName: \"kubernetes.io/projected/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-kube-api-access-fcfhh\") pod \"payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7\" (UID: \"8ecc4598-19ce-463b-a3de-5ee6ba41a08f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"
Nov 02 15:24:27.220386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.210437    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7390b194-bcc2-4020-98c4-73cfb8d07480-plugins-config-volume\") pod \"payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r\" (UID: \"7390b194-bcc2-4020-98c4-73cfb8d07480\") " pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"
Nov 02 15:24:27.220386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.211579    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7390b194-bcc2-4020-98c4-73cfb8d07480-plugins-config-volume\") pod \"payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r\" (UID: \"7390b194-bcc2-4020-98c4-73cfb8d07480\") " pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"
Nov 02 15:24:27.237024 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.236974    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"]
Nov 02 15:24:27.282618 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.282585    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkgnl\" (UniqueName: \"kubernetes.io/projected/7390b194-bcc2-4020-98c4-73cfb8d07480-kube-api-access-hkgnl\") pod \"payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r\" (UID: \"7390b194-bcc2-4020-98c4-73cfb8d07480\") " pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"
Nov 02 15:24:27.393568 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.318754    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7\" (UID: \"8ecc4598-19ce-463b-a3de-5ee6ba41a08f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"
Nov 02 15:24:27.393568 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.318822    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcfhh\" (UniqueName: \"kubernetes.io/projected/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-kube-api-access-fcfhh\") pod \"payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7\" (UID: \"8ecc4598-19ce-463b-a3de-5ee6ba41a08f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"
Nov 02 15:24:27.393568 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.320330    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7\" (UID: \"8ecc4598-19ce-463b-a3de-5ee6ba41a08f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"
Nov 02 15:24:27.393568 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.338633    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"
Nov 02 15:24:27.405854 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.405258    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcfhh\" (UniqueName: \"kubernetes.io/projected/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-kube-api-access-fcfhh\") pod \"payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7\" (UID: \"8ecc4598-19ce-463b-a3de-5ee6ba41a08f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"
Nov 02 15:24:27.496025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.495657    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"
Nov 02 15:24:27.650412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.646802    4891 generic.go:356] "Generic (PLEG): container finished" podID="830e624c-98fb-48c6-a9b1-e798013a7392" containerID="a258eafe36a27577be13f6cd49ce0a6632e27e31bcb1cc2cf08e8a550fe8fdc2" exitCode=0
Nov 02 15:24:27.650412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.647666    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w" event={"ID":"830e624c-98fb-48c6-a9b1-e798013a7392","Type":"ContainerDied","Data":"a258eafe36a27577be13f6cd49ce0a6632e27e31bcb1cc2cf08e8a550fe8fdc2"}
Nov 02 15:24:27.822060 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:24:27.822018    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7390b194_bcc2_4020_98c4_73cfb8d07480.slice/crio-0f2aea98457fb8113f6875ba2e1bd9471555540b5e4f5c837d8e985226c1c246 WatchSource:0}: Error finding container 0f2aea98457fb8113f6875ba2e1bd9471555540b5e4f5c837d8e985226c1c246: Status 404 returned error can't find the container with id 0f2aea98457fb8113f6875ba2e1bd9471555540b5e4f5c837d8e985226c1c246
Nov 02 15:24:27.824322 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.824279    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"]
Nov 02 15:24:27.855542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.855517    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:27.935937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.935906    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"]
Nov 02 15:24:27.940205 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:24:27.940169    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecc4598_19ce_463b_a3de_5ee6ba41a08f.slice/crio-2ff3d49a6a61557695260d785ffe9bd3d6cb00a4dbc2d1f8092b3f22ae9e0645 WatchSource:0}: Error finding container 2ff3d49a6a61557695260d785ffe9bd3d6cb00a4dbc2d1f8092b3f22ae9e0645: Status 404 returned error can't find the container with id 2ff3d49a6a61557695260d785ffe9bd3d6cb00a4dbc2d1f8092b3f22ae9e0645
Nov 02 15:24:27.945232 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.945203    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-openshift-ca\") pod \"25b284eb-d840-4103-9ef8-7115157d7333\" (UID: \"25b284eb-d840-4103-9ef8-7115157d7333\") "
Nov 02 15:24:27.945533 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.945513    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b284eb-d840-4103-9ef8-7115157d7333-kube-api-access-qjnpk\" (UniqueName: \"kubernetes.io/projected/25b284eb-d840-4103-9ef8-7115157d7333-kube-api-access-qjnpk\") pod \"25b284eb-d840-4103-9ef8-7115157d7333\" (UID: \"25b284eb-d840-4103-9ef8-7115157d7333\") "
Nov 02 15:24:27.945660 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.945644    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-xks-ca\") pod \"25b284eb-d840-4103-9ef8-7115157d7333\" (UID: \"25b284eb-d840-4103-9ef8-7115157d7333\") "
Nov 02 15:24:27.945732 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.945698    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-openshift-ca" pod "25b284eb-d840-4103-9ef8-7115157d7333" (UID: "25b284eb-d840-4103-9ef8-7115157d7333"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:24:27.946334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.946297    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:24:27.948260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.948228    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-xks-ca" pod "25b284eb-d840-4103-9ef8-7115157d7333" (UID: "25b284eb-d840-4103-9ef8-7115157d7333"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:24:27.948260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:27.948242    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b284eb-d840-4103-9ef8-7115157d7333-kube-api-access-qjnpk" pod "25b284eb-d840-4103-9ef8-7115157d7333" (UID: "25b284eb-d840-4103-9ef8-7115157d7333"). InnerVolumeSpecName "kube-api-access-qjnpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:24:28.047825 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.047799    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjnpk\" (UniqueName: \"kubernetes.io/projected/25b284eb-d840-4103-9ef8-7115157d7333-kube-api-access-qjnpk\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:24:28.047825 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.047827    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b284eb-d840-4103-9ef8-7115157d7333-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:24:28.656964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.656939    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"
Nov 02 15:24:28.657387 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.656938    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8" event={"ID":"25b284eb-d840-4103-9ef8-7115157d7333","Type":"ContainerDied","Data":"7330a76d07b75f30bf707eec76e5b73829e31f9319dcc787128ea27aa3fe5c70"}
Nov 02 15:24:28.657387 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.657070    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7330a76d07b75f30bf707eec76e5b73829e31f9319dcc787128ea27aa3fe5c70"
Nov 02 15:24:28.659050 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.659010    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7" event={"ID":"8ecc4598-19ce-463b-a3de-5ee6ba41a08f","Type":"ContainerStarted","Data":"c0fb0e54f4541765d9fad7748a1e56e1f7ff63b36422f6124fd038189948f221"}
Nov 02 15:24:28.659216 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.659062    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7" event={"ID":"8ecc4598-19ce-463b-a3de-5ee6ba41a08f","Type":"ContainerStarted","Data":"2ff3d49a6a61557695260d785ffe9bd3d6cb00a4dbc2d1f8092b3f22ae9e0645"}
Nov 02 15:24:28.659216 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.659119    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"
Nov 02 15:24:28.660909 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.660881    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r" event={"ID":"7390b194-bcc2-4020-98c4-73cfb8d07480","Type":"ContainerStarted","Data":"84b625e3d4ddb407046a35ddbecbb18be2ef00c46abe04b81b1b2fa67fd573f3"}
Nov 02 15:24:28.661013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.660925    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r" event={"ID":"7390b194-bcc2-4020-98c4-73cfb8d07480","Type":"ContainerStarted","Data":"0f2aea98457fb8113f6875ba2e1bd9471555540b5e4f5c837d8e985226c1c246"}
Nov 02 15:24:28.661068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.661036    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"
Nov 02 15:24:28.813557 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.813508    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7" podStartSLOduration=1.813493078 podStartE2EDuration="1.813493078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:28.808977844 +0000 UTC m=+1676.401594645" watchObservedRunningTime="2026-11-02 15:24:28.813493078 +0000 UTC m=+1676.406109871"
Nov 02 15:24:28.845956 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.845930    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:28.961755 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.961727    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830e624c-98fb-48c6-a9b1-e798013a7392-kube-api-access-jd2bq\" (UniqueName: \"kubernetes.io/projected/830e624c-98fb-48c6-a9b1-e798013a7392-kube-api-access-jd2bq\") pod \"830e624c-98fb-48c6-a9b1-e798013a7392\" (UID: \"830e624c-98fb-48c6-a9b1-e798013a7392\") "
Nov 02 15:24:28.961932 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.961770    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-openshift-ca\") pod \"830e624c-98fb-48c6-a9b1-e798013a7392\" (UID: \"830e624c-98fb-48c6-a9b1-e798013a7392\") "
Nov 02 15:24:28.961932 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.961809    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-xks-ca\") pod \"830e624c-98fb-48c6-a9b1-e798013a7392\" (UID: \"830e624c-98fb-48c6-a9b1-e798013a7392\") "
Nov 02 15:24:28.962194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.962169    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-openshift-ca" pod "830e624c-98fb-48c6-a9b1-e798013a7392" (UID: "830e624c-98fb-48c6-a9b1-e798013a7392"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:24:28.962754 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.962727    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:24:28.963957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.963933    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830e624c-98fb-48c6-a9b1-e798013a7392-kube-api-access-jd2bq" pod "830e624c-98fb-48c6-a9b1-e798013a7392" (UID: "830e624c-98fb-48c6-a9b1-e798013a7392"). InnerVolumeSpecName "kube-api-access-jd2bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:24:28.964047 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.963963    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-xks-ca" pod "830e624c-98fb-48c6-a9b1-e798013a7392" (UID: "830e624c-98fb-48c6-a9b1-e798013a7392"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:24:28.983459 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:28.983414    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r" podStartSLOduration=2.98340089 podStartE2EDuration="2.98340089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:28.976873699 +0000 UTC m=+1676.569490489" watchObservedRunningTime="2026-11-02 15:24:28.98340089 +0000 UTC m=+1676.576017704"
Nov 02 15:24:29.064391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:29.064358    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd2bq\" (UniqueName: \"kubernetes.io/projected/830e624c-98fb-48c6-a9b1-e798013a7392-kube-api-access-jd2bq\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:24:29.064391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:29.064388    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830e624c-98fb-48c6-a9b1-e798013a7392-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:24:29.672860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:29.672832    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"
Nov 02 15:24:29.673364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:29.672861    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w" event={"ID":"830e624c-98fb-48c6-a9b1-e798013a7392","Type":"ContainerDied","Data":"00aaeeef5b2f267cbe7327042941b593618196c3fdfe9e42903edb16155ca112"}
Nov 02 15:24:29.673364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:29.672911    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00aaeeef5b2f267cbe7327042941b593618196c3fdfe9e42903edb16155ca112"
Nov 02 15:24:34.412345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.412305    4891 scope.go:122] "RemoveContainer" containerID="55215f96bda58e4c52bba1c647274a4df5ac7d5eb4a86478456f025f76af32f4"
Nov 02 15:24:34.423925 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.423899    4891 scope.go:122] "RemoveContainer" containerID="cd83abc56a3f7280b3f6561632ca1dacbfdcacc663a18e783f854889219654af"
Nov 02 15:24:34.434193 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.434172    4891 scope.go:122] "RemoveContainer" containerID="c91c5e3441a82c726b1921dd23af53fa28a6850d7296fad4e6e0766d98430d9f"
Nov 02 15:24:34.444290 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.444271    4891 scope.go:122] "RemoveContainer" containerID="bd19a1490f1f1c789a613e9ed781d11cdf76eb6001d66de59faf7556bb91f53c"
Nov 02 15:24:34.456094 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.456075    4891 scope.go:122] "RemoveContainer" containerID="b800bc7940d521e2aa4a45f64fff501f916bc8a243b3442c0e5470be188f091f"
Nov 02 15:24:34.466748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.466731    4891 scope.go:122] "RemoveContainer" containerID="913cada4a089ecb7d8494d2364e9c3936cde039429bd9350064f7f45793b8185"
Nov 02 15:24:34.476414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.476394    4891 scope.go:122] "RemoveContainer" containerID="628d683ecc36b743d9ad3c9cc9ea7818119a5ce05247eb5dd90ef71447602085"
Nov 02 15:24:34.486198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.486173    4891 scope.go:122] "RemoveContainer" containerID="0479f2ce8dc89e852960cf0726ed2b762671e67a4d1c3f87404dab76c6e5e3bd"
Nov 02 15:24:34.499540 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.499514    4891 scope.go:122] "RemoveContainer" containerID="a256977d0a4a3b9858d2e42e6025ee517e58fa3a29945bce852ada11933e3bdd"
Nov 02 15:24:34.523583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.523553    4891 scope.go:122] "RemoveContainer" containerID="ccb9b3aa14c666b54d02f96fa02e1342745bc158ef3101aa19f1e4736308679f"
Nov 02 15:24:34.539251 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.539232    4891 scope.go:122] "RemoveContainer" containerID="4ac8530de89407dad301a5ae100ca38045c1ccf9ca8ba1431ed53e6f68537695"
Nov 02 15:24:34.548960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.548938    4891 scope.go:122] "RemoveContainer" containerID="cad94c142c3403d0dc32949e602b6d228fc6aca6b97c1be7d2d72076a49962f4"
Nov 02 15:24:34.558684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:34.558665    4891 scope.go:122] "RemoveContainer" containerID="8ef89077a59fc1098347ec243606b9c570893610d74ceb8b4c1acad76a6787d5"
Nov 02 15:24:39.676703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:39.676674    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"
Nov 02 15:24:39.677271 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:24:39.676833    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"
Nov 02 15:24:44.282387 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:24:44.282344    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:25:04.013287 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:04.013249    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"]
Nov 02 15:25:04.062300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:04.062263    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-be5d503e-k2pn8"]
Nov 02 15:25:04.527581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:04.527541    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b284eb-d840-4103-9ef8-7115157d7333" path="/var/lib/kubelet/pods/25b284eb-d840-4103-9ef8-7115157d7333/volumes"
Nov 02 15:25:06.379937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:06.379164    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:25:06.449551 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:06.449503    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:25:06.454796 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:06.454072    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:25:08.357562 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:08.357526    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"]
Nov 02 15:25:08.358008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:08.357756    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" podUID="04eab7d2-8782-4353-b7b3-1ea5df096979" containerName="istio-proxy" containerID="cri-o://84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c" gracePeriod=30
Nov 02 15:25:08.407394 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:08.407357    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"]
Nov 02 15:25:08.407665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:08.407611    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7" podUID="1559e94a-36ed-40a4-8118-3d8f0f338a17" containerName="istio-proxy" containerID="cri-o://77c95b8bc043509e7aebb8232ffadb7e8f2709821949f66c497af131683c7480" gracePeriod=30
Nov 02 15:25:08.412547 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:08.412519    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"]
Nov 02 15:25:08.412728 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:08.412708    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt" podUID="dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" containerName="istio-proxy" containerID="cri-o://db8f9da6aa43f6c558b03328af65bc14936442e841d87c0aebe3159bf432d7a1" gracePeriod=30
Nov 02 15:25:08.460187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:08.460153    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"]
Nov 02 15:25:08.460624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:08.460594    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" podUID="d2a3b4a5-282a-473b-8d71-287a475429b1" containerName="istio-proxy" containerID="cri-o://69e1b4d604670ad26bb4a634ab5fab935a414304d303a771410e68878e640010" gracePeriod=30
Nov 02 15:25:13.727687 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.727287    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:25:13.764418 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.764354    4891 generic.go:356] "Generic (PLEG): container finished" podID="1559e94a-36ed-40a4-8118-3d8f0f338a17" containerID="77c95b8bc043509e7aebb8232ffadb7e8f2709821949f66c497af131683c7480" exitCode=0
Nov 02 15:25:13.764618 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.764591    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7" event={"ID":"1559e94a-36ed-40a4-8118-3d8f0f338a17","Type":"ContainerDied","Data":"77c95b8bc043509e7aebb8232ffadb7e8f2709821949f66c497af131683c7480"}
Nov 02 15:25:13.768388 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.766760    4891 generic.go:356] "Generic (PLEG): container finished" podID="dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" containerID="db8f9da6aa43f6c558b03328af65bc14936442e841d87c0aebe3159bf432d7a1" exitCode=0
Nov 02 15:25:13.768388 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.766883    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt" event={"ID":"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768","Type":"ContainerDied","Data":"db8f9da6aa43f6c558b03328af65bc14936442e841d87c0aebe3159bf432d7a1"}
Nov 02 15:25:13.770151 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.770125    4891 generic.go:356] "Generic (PLEG): container finished" podID="d2a3b4a5-282a-473b-8d71-287a475429b1" containerID="69e1b4d604670ad26bb4a634ab5fab935a414304d303a771410e68878e640010" exitCode=0
Nov 02 15:25:13.770302 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.770235    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" event={"ID":"d2a3b4a5-282a-473b-8d71-287a475429b1","Type":"ContainerDied","Data":"69e1b4d604670ad26bb4a634ab5fab935a414304d303a771410e68878e640010"}
Nov 02 15:25:13.771884 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.771864    4891 generic.go:356] "Generic (PLEG): container finished" podID="04eab7d2-8782-4353-b7b3-1ea5df096979" containerID="84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c" exitCode=0
Nov 02 15:25:13.771972 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.771898    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" event={"ID":"04eab7d2-8782-4353-b7b3-1ea5df096979","Type":"ContainerDied","Data":"84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c"}
Nov 02 15:25:13.771972 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.771923    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"
Nov 02 15:25:13.771972 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.771946    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj" event={"ID":"04eab7d2-8782-4353-b7b3-1ea5df096979","Type":"ContainerDied","Data":"efe5d80630bd37cb74fc18d1b8f1b13c6562093df575df2c4a518145afa556ed"}
Nov 02 15:25:13.771972 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.771964    4891 scope.go:122] "RemoveContainer" containerID="84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c"
Nov 02 15:25:13.785650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.785629    4891 scope.go:122] "RemoveContainer" containerID="84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c"
Nov 02 15:25:13.786007 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:25:13.785982    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c\": container with ID starting with 84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c not found: ID does not exist" containerID="84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c"
Nov 02 15:25:13.786116 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.786018    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c"} err="failed to get container status \"84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c\": rpc error: code = NotFound desc = could not find container \"84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c\": container with ID starting with 84d033d708a099c83b4b27172fe7c86dfd76ccf768dc5595c9e4c1608d251f1c not found: ID does not exist"
Nov 02 15:25:13.803381 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.803358    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:25:13.805539 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.805514    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-socket\") pod \"04eab7d2-8782-4353-b7b3-1ea5df096979\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") "
Nov 02 15:25:13.805881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.805676    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04eab7d2-8782-4353-b7b3-1ea5df096979-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04eab7d2-8782-4353-b7b3-1ea5df096979-istiod-ca-cert\") pod \"04eab7d2-8782-4353-b7b3-1ea5df096979\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") "
Nov 02 15:25:13.805881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.805711    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-data\") pod \"04eab7d2-8782-4353-b7b3-1ea5df096979\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") "
Nov 02 15:25:13.805881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.805770    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-socket" pod "04eab7d2-8782-4353-b7b3-1ea5df096979" (UID: "04eab7d2-8782-4353-b7b3-1ea5df096979"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.805881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.805822    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-podinfo\") pod \"04eab7d2-8782-4353-b7b3-1ea5df096979\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") "
Nov 02 15:25:13.805881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.805880    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-credential-socket\") pod \"04eab7d2-8782-4353-b7b3-1ea5df096979\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") "
Nov 02 15:25:13.806209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.805909    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-token\" (UniqueName: \"kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-token\") pod \"04eab7d2-8782-4353-b7b3-1ea5df096979\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") "
Nov 02 15:25:13.806209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.805990    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-kube-api-access-rpxw7\" (UniqueName: \"kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-kube-api-access-rpxw7\") pod \"04eab7d2-8782-4353-b7b3-1ea5df096979\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") "
Nov 02 15:25:13.806209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.806047    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-certs\") pod \"04eab7d2-8782-4353-b7b3-1ea5df096979\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") "
Nov 02 15:25:13.806209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.806111    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-data" pod "04eab7d2-8782-4353-b7b3-1ea5df096979" (UID: "04eab7d2-8782-4353-b7b3-1ea5df096979"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.806209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.806126    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-envoy\") pod \"04eab7d2-8782-4353-b7b3-1ea5df096979\" (UID: \"04eab7d2-8782-4353-b7b3-1ea5df096979\") "
Nov 02 15:25:13.807179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.806911    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-certs" pod "04eab7d2-8782-4353-b7b3-1ea5df096979" (UID: "04eab7d2-8782-4353-b7b3-1ea5df096979"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.807179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.806941    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.807179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.806964    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.807586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.807550    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-credential-socket" pod "04eab7d2-8782-4353-b7b3-1ea5df096979" (UID: "04eab7d2-8782-4353-b7b3-1ea5df096979"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.807834 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.807805    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04eab7d2-8782-4353-b7b3-1ea5df096979-istiod-ca-cert" pod "04eab7d2-8782-4353-b7b3-1ea5df096979" (UID: "04eab7d2-8782-4353-b7b3-1ea5df096979"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:13.809543 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.809519    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-podinfo" pod "04eab7d2-8782-4353-b7b3-1ea5df096979" (UID: "04eab7d2-8782-4353-b7b3-1ea5df096979"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:25:13.810788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.810764    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-token" pod "04eab7d2-8782-4353-b7b3-1ea5df096979" (UID: "04eab7d2-8782-4353-b7b3-1ea5df096979"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:13.810886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.810776    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-envoy" pod "04eab7d2-8782-4353-b7b3-1ea5df096979" (UID: "04eab7d2-8782-4353-b7b3-1ea5df096979"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.811992 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.811969    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-kube-api-access-rpxw7" pod "04eab7d2-8782-4353-b7b3-1ea5df096979" (UID: "04eab7d2-8782-4353-b7b3-1ea5df096979"). InnerVolumeSpecName "kube-api-access-rpxw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:13.851812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.851759    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:25:13.907824 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.907792    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-certs\") pod \"d2a3b4a5-282a-473b-8d71-287a475429b1\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") "
Nov 02 15:25:13.908008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.907864    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1559e94a-36ed-40a4-8118-3d8f0f338a17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1559e94a-36ed-40a4-8118-3d8f0f338a17-istiod-ca-cert\") pod \"1559e94a-36ed-40a4-8118-3d8f0f338a17\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") "
Nov 02 15:25:13.908008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.907916    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-token\" (UniqueName: \"kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-token\") pod \"d2a3b4a5-282a-473b-8d71-287a475429b1\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") "
Nov 02 15:25:13.908177 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908008    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-data\") pod \"1559e94a-36ed-40a4-8118-3d8f0f338a17\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") "
Nov 02 15:25:13.908177 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908039    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-certs" pod "d2a3b4a5-282a-473b-8d71-287a475429b1" (UID: "d2a3b4a5-282a-473b-8d71-287a475429b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.908177 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908113    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-envoy\") pod \"d2a3b4a5-282a-473b-8d71-287a475429b1\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") "
Nov 02 15:25:13.908177 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908161    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-data\") pod \"d2a3b4a5-282a-473b-8d71-287a475429b1\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") "
Nov 02 15:25:13.908349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908200    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-credential-socket\") pod \"1559e94a-36ed-40a4-8118-3d8f0f338a17\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") "
Nov 02 15:25:13.908349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908225    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-socket\") pod \"d2a3b4a5-282a-473b-8d71-287a475429b1\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") "
Nov 02 15:25:13.908349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908280    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-podinfo\") pod \"d2a3b4a5-282a-473b-8d71-287a475429b1\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") "
Nov 02 15:25:13.908481 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908351    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-podinfo\") pod \"1559e94a-36ed-40a4-8118-3d8f0f338a17\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") "
Nov 02 15:25:13.908481 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908363    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1559e94a-36ed-40a4-8118-3d8f0f338a17-istiod-ca-cert" pod "1559e94a-36ed-40a4-8118-3d8f0f338a17" (UID: "1559e94a-36ed-40a4-8118-3d8f0f338a17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:13.908481 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908420    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-socket\") pod \"1559e94a-36ed-40a4-8118-3d8f0f338a17\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") "
Nov 02 15:25:13.908481 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908469    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a3b4a5-282a-473b-8d71-287a475429b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a3b4a5-282a-473b-8d71-287a475429b1-istiod-ca-cert\") pod \"d2a3b4a5-282a-473b-8d71-287a475429b1\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") "
Nov 02 15:25:13.908663 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908554    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-kube-api-access-tw8dx\" (UniqueName: \"kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-kube-api-access-tw8dx\") pod \"1559e94a-36ed-40a4-8118-3d8f0f338a17\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") "
Nov 02 15:25:13.908663 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908593    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-data" pod "d2a3b4a5-282a-473b-8d71-287a475429b1" (UID: "d2a3b4a5-282a-473b-8d71-287a475429b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.908663 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908612    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-credential-socket\") pod \"d2a3b4a5-282a-473b-8d71-287a475429b1\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") "
Nov 02 15:25:13.908663 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908652    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-token\" (UniqueName: \"kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-token\") pod \"1559e94a-36ed-40a4-8118-3d8f0f338a17\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") "
Nov 02 15:25:13.908853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908728    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-kube-api-access-knrrr\" (UniqueName: \"kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-kube-api-access-knrrr\") pod \"d2a3b4a5-282a-473b-8d71-287a475429b1\" (UID: \"d2a3b4a5-282a-473b-8d71-287a475429b1\") "
Nov 02 15:25:13.908853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908784    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-certs\") pod \"1559e94a-36ed-40a4-8118-3d8f0f338a17\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") "
Nov 02 15:25:13.908853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.908818    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-envoy\") pod \"1559e94a-36ed-40a4-8118-3d8f0f338a17\" (UID: \"1559e94a-36ed-40a4-8118-3d8f0f338a17\") "
Nov 02 15:25:13.909698 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909498    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-socket" pod "d2a3b4a5-282a-473b-8d71-287a475429b1" (UID: "d2a3b4a5-282a-473b-8d71-287a475429b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.909698 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909516    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-credential-socket" pod "1559e94a-36ed-40a4-8118-3d8f0f338a17" (UID: "1559e94a-36ed-40a4-8118-3d8f0f338a17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909729    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909753    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909769    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpxw7\" (UniqueName: \"kubernetes.io/projected/04eab7d2-8782-4353-b7b3-1ea5df096979-kube-api-access-rpxw7\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909785    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909799    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909814    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909828    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1559e94a-36ed-40a4-8118-3d8f0f338a17-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909843    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04eab7d2-8782-4353-b7b3-1ea5df096979-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909858    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909872    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909886    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.909901    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04eab7d2-8782-4353-b7b3-1ea5df096979-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:13.910221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.910224    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-data" pod "1559e94a-36ed-40a4-8118-3d8f0f338a17" (UID: "1559e94a-36ed-40a4-8118-3d8f0f338a17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.911039 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.911010    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-socket" pod "1559e94a-36ed-40a4-8118-3d8f0f338a17" (UID: "1559e94a-36ed-40a4-8118-3d8f0f338a17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.911602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.911417    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a3b4a5-282a-473b-8d71-287a475429b1-istiod-ca-cert" pod "d2a3b4a5-282a-473b-8d71-287a475429b1" (UID: "d2a3b4a5-282a-473b-8d71-287a475429b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:13.913415 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.913189    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-credential-socket" pod "d2a3b4a5-282a-473b-8d71-287a475429b1" (UID: "d2a3b4a5-282a-473b-8d71-287a475429b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.913722 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.913436    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-podinfo" pod "d2a3b4a5-282a-473b-8d71-287a475429b1" (UID: "d2a3b4a5-282a-473b-8d71-287a475429b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:25:13.913895 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.913859    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-certs" pod "1559e94a-36ed-40a4-8118-3d8f0f338a17" (UID: "1559e94a-36ed-40a4-8118-3d8f0f338a17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.915327 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.915261    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:25:13.918115 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.918073    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-kube-api-access-knrrr" pod "d2a3b4a5-282a-473b-8d71-287a475429b1" (UID: "d2a3b4a5-282a-473b-8d71-287a475429b1"). InnerVolumeSpecName "kube-api-access-knrrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:13.918979 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.918957    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-podinfo" pod "1559e94a-36ed-40a4-8118-3d8f0f338a17" (UID: "1559e94a-36ed-40a4-8118-3d8f0f338a17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:25:13.920272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.920244    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-envoy" pod "d2a3b4a5-282a-473b-8d71-287a475429b1" (UID: "d2a3b4a5-282a-473b-8d71-287a475429b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.920752 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.920731    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-token" pod "1559e94a-36ed-40a4-8118-3d8f0f338a17" (UID: "1559e94a-36ed-40a4-8118-3d8f0f338a17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:13.920886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.920738    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-envoy" pod "1559e94a-36ed-40a4-8118-3d8f0f338a17" (UID: "1559e94a-36ed-40a4-8118-3d8f0f338a17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:13.920886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.920764    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-kube-api-access-tw8dx" pod "1559e94a-36ed-40a4-8118-3d8f0f338a17" (UID: "1559e94a-36ed-40a4-8118-3d8f0f338a17"). InnerVolumeSpecName "kube-api-access-tw8dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:13.933126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:13.933072    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-token" pod "d2a3b4a5-282a-473b-8d71-287a475429b1" (UID: "d2a3b4a5-282a-473b-8d71-287a475429b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:14.010939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.010901    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-envoy\") pod \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") "
Nov 02 15:25:14.011137 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.010970    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istiod-ca-cert\") pod \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") "
Nov 02 15:25:14.011137 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.011110    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-data\") pod \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") "
Nov 02 15:25:14.011266 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.011151    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-token\" (UniqueName: \"kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-token\") pod \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") "
Nov 02 15:25:14.011266 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.011248    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-certs\") pod \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") "
Nov 02 15:25:14.011393 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.011368    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-podinfo\") pod \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") "
Nov 02 15:25:14.011452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.011413    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-socket\") pod \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") "
Nov 02 15:25:14.012061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.011549    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-kube-api-access-brc8v\" (UniqueName: \"kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-kube-api-access-brc8v\") pod \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") "
Nov 02 15:25:14.012061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.011656    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-credential-socket\") pod \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\" (UID: \"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768\") "
Nov 02 15:25:14.012061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.011712    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-certs" pod "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" (UID: "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:14.012061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.011919    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-socket" pod "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" (UID: "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:14.012357 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012305    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-credential-socket" pod "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" (UID: "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:14.012464 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012428    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.012464 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012450    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.012591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012465    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.012591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012480    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a3b4a5-282a-473b-8d71-287a475429b1-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.012591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012495    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw8dx\" (UniqueName: \"kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-kube-api-access-tw8dx\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.012591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012509    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.012591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012523    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.012591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012536    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knrrr\" (UniqueName: \"kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-kube-api-access-knrrr\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.012591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012553    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.012591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012567    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.012591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012580    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.012591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012594    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1559e94a-36ed-40a4-8118-3d8f0f338a17-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.013126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012608    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.013126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012620    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a3b4a5-282a-473b-8d71-287a475429b1-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.013126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012633    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.013126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012706    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-data" pod "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" (UID: "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:14.013126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.012861    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istiod-ca-cert" pod "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" (UID: "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:14.014283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.014254    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-podinfo" pod "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" (UID: "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:25:14.014864 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.014828    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-envoy" pod "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" (UID: "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:14.014954 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.014864    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-kube-api-access-brc8v" pod "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" (UID: "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768"). InnerVolumeSpecName "kube-api-access-brc8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:14.015373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.015287    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-token" pod "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" (UID: "dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:14.158049 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.157987    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.158049 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.158040    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.158049 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.158054    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.158377 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.158071    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brc8v\" (UniqueName: \"kubernetes.io/projected/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-kube-api-access-brc8v\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.158377 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.158087    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.158377 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.158105    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.158377 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.158119    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:25:14.275084 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:25:14.275051    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:25:14.405640 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.405608    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"]
Nov 02 15:25:14.410234 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.410210    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tcvnj"]
Nov 02 15:25:14.523340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.523289    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04eab7d2-8782-4353-b7b3-1ea5df096979" path="/var/lib/kubelet/pods/04eab7d2-8782-4353-b7b3-1ea5df096979/volumes"
Nov 02 15:25:14.782092 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.782060    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"
Nov 02 15:25:14.782598 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.782061    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt" event={"ID":"dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768","Type":"ContainerDied","Data":"e51d08164547190bdec520291443d7e2689a6e2a013dcbddff1315d9fafcda56"}
Nov 02 15:25:14.782598 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.782214    4891 scope.go:122] "RemoveContainer" containerID="db8f9da6aa43f6c558b03328af65bc14936442e841d87c0aebe3159bf432d7a1"
Nov 02 15:25:14.783694 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.783639    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"
Nov 02 15:25:14.783694 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.783676    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69" event={"ID":"d2a3b4a5-282a-473b-8d71-287a475429b1","Type":"ContainerDied","Data":"4b3fab7c4912219fbb90652988f3bd0170b9134a2d7173b910d5934cc5b81ece"}
Nov 02 15:25:14.786709 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.786685    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7" event={"ID":"1559e94a-36ed-40a4-8118-3d8f0f338a17","Type":"ContainerDied","Data":"18e48912dcc6a04e997f5927ac7cfa7301ba67f956bfac1d576247e315e4f657"}
Nov 02 15:25:14.786709 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.786719    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"
Nov 02 15:25:14.794428 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.794406    4891 scope.go:122] "RemoveContainer" containerID="69e1b4d604670ad26bb4a634ab5fab935a414304d303a771410e68878e640010"
Nov 02 15:25:14.804411 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.804393    4891 scope.go:122] "RemoveContainer" containerID="77c95b8bc043509e7aebb8232ffadb7e8f2709821949f66c497af131683c7480"
Nov 02 15:25:14.949617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.949579    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"]
Nov 02 15:25:14.953967 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:14.953940    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-tfln7"]
Nov 02 15:25:15.063883 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:15.063801    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"]
Nov 02 15:25:15.068501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:15.068478    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-zr4vt"]
Nov 02 15:25:15.177541 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:15.177501    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"]
Nov 02 15:25:15.181678 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:15.181653    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-42555d-openshift-default-6c8c6d89c-njj69"]
Nov 02 15:25:16.524538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:16.524501    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1559e94a-36ed-40a4-8118-3d8f0f338a17" path="/var/lib/kubelet/pods/1559e94a-36ed-40a4-8118-3d8f0f338a17/volumes"
Nov 02 15:25:16.525117 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:16.525100    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a3b4a5-282a-473b-8d71-287a475429b1" path="/var/lib/kubelet/pods/d2a3b4a5-282a-473b-8d71-287a475429b1/volumes"
Nov 02 15:25:16.525696 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:16.525678    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" path="/var/lib/kubelet/pods/dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768/volumes"
Nov 02 15:25:29.051166 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:29.051134    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"]
Nov 02 15:25:29.100475 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:29.100441    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20b67a2-q2d95"]
Nov 02 15:25:30.525716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:30.525675    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995af847-3a60-4464-910f-d8d66dbc3672" path="/var/lib/kubelet/pods/995af847-3a60-4464-910f-d8d66dbc3672/volumes"
Nov 02 15:25:34.656786 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:34.656732    4891 scope.go:122] "RemoveContainer" containerID="65f4fe656a15ba8cdaa8b2a921c828434e26a3d9befc7b32eb9b976406fc19b0"
Nov 02 15:25:34.670797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:34.670772    4891 scope.go:122] "RemoveContainer" containerID="3d5df48263216c255554688df21e881655fb35334a271eb97c271a3d7dc80100"
Nov 02 15:25:34.681219 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:34.681198    4891 scope.go:122] "RemoveContainer" containerID="9d53a45be22c572f052eef4ea7fb86d3bfbf1c0019833b4ba01abe7784689e5a"
Nov 02 15:25:34.690651 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:25:34.690629    4891 scope.go:122] "RemoveContainer" containerID="7e0341744c87ca93f82e815a1e0b5687f50c643cb9a4e01fc7199792be3659d0"
Nov 02 15:25:44.284532 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:25:44.275430    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:26:14.274026 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:26:14.273988    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:26:33.113457 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:33.113428    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:26:33.114638 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:33.114615    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:26:33.115053 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:33.115029    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:26:33.115959 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:33.115938    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:26:33.129490 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:33.129467    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:26:33.129949 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:33.129935    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:26:34.764045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:34.764011    4891 scope.go:122] "RemoveContainer" containerID="7c1db10e74eedc1efa19da8464d1788260f7f6e62abeb0ffa196dee588e82e98"
Nov 02 15:26:34.774732 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:34.774711    4891 scope.go:122] "RemoveContainer" containerID="b60782196ad61539c1113d20cf729c01fdf598386c1401bf45c04082cc0e5e66"
Nov 02 15:26:34.784257 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:34.784237    4891 scope.go:122] "RemoveContainer" containerID="e0bc94bada53baa514935114fde5dd5caae6ce25601fd16ca0004854c62107fe"
Nov 02 15:26:34.793559 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:34.793538    4891 scope.go:122] "RemoveContainer" containerID="d0845fe43f4f8b84dc43d1990f795bc63f00b28cfa104767ea1d0b33178c73c9"
Nov 02 15:26:39.034236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:39.034197    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"]
Nov 02 15:26:39.034781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:39.034428    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7" podUID="014837c5-3da0-4c7d-a46a-92a84dac45fc" containerName="istio-proxy" containerID="cri-o://515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2" gracePeriod=30
Nov 02 15:26:39.154574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:39.154535    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"]
Nov 02 15:26:39.154809 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:39.154783    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy" containerID="cri-o://0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9" gracePeriod=30
Nov 02 15:26:40.485052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:40.485011    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:26:40.485532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:40.485080    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7" podUID="014837c5-3da0-4c7d-a46a-92a84dac45fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:26:44.284842 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:26:44.284797    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:26:44.364885 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.364855    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:26:44.522373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.509415    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:26:44.534779 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.534745    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-socket\") pod \"014837c5-3da0-4c7d-a46a-92a84dac45fc\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") "
Nov 02 15:26:44.534955 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.534813    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-token\" (UniqueName: \"kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-token\") pod \"014837c5-3da0-4c7d-a46a-92a84dac45fc\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") "
Nov 02 15:26:44.534955 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.534868    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-kube-api-access-slvmx\" (UniqueName: \"kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-kube-api-access-slvmx\") pod \"014837c5-3da0-4c7d-a46a-92a84dac45fc\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") "
Nov 02 15:26:44.534955 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.534915    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-data\") pod \"014837c5-3da0-4c7d-a46a-92a84dac45fc\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") "
Nov 02 15:26:44.535122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.534964    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-podinfo\") pod \"014837c5-3da0-4c7d-a46a-92a84dac45fc\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") "
Nov 02 15:26:44.535122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.535113    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-envoy\") pod \"014837c5-3da0-4c7d-a46a-92a84dac45fc\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") "
Nov 02 15:26:44.535599 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.535561    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-socket" pod "014837c5-3da0-4c7d-a46a-92a84dac45fc" (UID: "014837c5-3da0-4c7d-a46a-92a84dac45fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:44.535695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.535664    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-certs\") pod \"014837c5-3da0-4c7d-a46a-92a84dac45fc\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") "
Nov 02 15:26:44.535751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.535713    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014837c5-3da0-4c7d-a46a-92a84dac45fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014837c5-3da0-4c7d-a46a-92a84dac45fc-istiod-ca-cert\") pod \"014837c5-3da0-4c7d-a46a-92a84dac45fc\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") "
Nov 02 15:26:44.535805 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.535794    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-credential-socket\") pod \"014837c5-3da0-4c7d-a46a-92a84dac45fc\" (UID: \"014837c5-3da0-4c7d-a46a-92a84dac45fc\") "
Nov 02 15:26:44.536393 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.536371    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.538866 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.538842    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-data" pod "014837c5-3da0-4c7d-a46a-92a84dac45fc" (UID: "014837c5-3da0-4c7d-a46a-92a84dac45fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:44.538979 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.538953    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014837c5-3da0-4c7d-a46a-92a84dac45fc-istiod-ca-cert" pod "014837c5-3da0-4c7d-a46a-92a84dac45fc" (UID: "014837c5-3da0-4c7d-a46a-92a84dac45fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:26:44.540886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.540857    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-certs" pod "014837c5-3da0-4c7d-a46a-92a84dac45fc" (UID: "014837c5-3da0-4c7d-a46a-92a84dac45fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:44.541092 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.541072    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-credential-socket" pod "014837c5-3da0-4c7d-a46a-92a84dac45fc" (UID: "014837c5-3da0-4c7d-a46a-92a84dac45fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:44.542528 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.542501    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-envoy" pod "014837c5-3da0-4c7d-a46a-92a84dac45fc" (UID: "014837c5-3da0-4c7d-a46a-92a84dac45fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:44.542654 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.542532    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-kube-api-access-slvmx" pod "014837c5-3da0-4c7d-a46a-92a84dac45fc" (UID: "014837c5-3da0-4c7d-a46a-92a84dac45fc"). InnerVolumeSpecName "kube-api-access-slvmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:26:44.543120 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.543100    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-podinfo" pod "014837c5-3da0-4c7d-a46a-92a84dac45fc" (UID: "014837c5-3da0-4c7d-a46a-92a84dac45fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:26:44.543788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.543770    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-token" pod "014837c5-3da0-4c7d-a46a-92a84dac45fc" (UID: "014837c5-3da0-4c7d-a46a-92a84dac45fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:26:44.636919 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.636826    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-data\") pod \"9d1828c6-62fc-497b-9ef9-e12d001f4683\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") "
Nov 02 15:26:44.636919 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.636906    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-certs\") pod \"9d1828c6-62fc-497b-9ef9-e12d001f4683\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") "
Nov 02 15:26:44.637175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.636935    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d1828c6-62fc-497b-9ef9-e12d001f4683-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1828c6-62fc-497b-9ef9-e12d001f4683-istiod-ca-cert\") pod \"9d1828c6-62fc-497b-9ef9-e12d001f4683\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") "
Nov 02 15:26:44.637175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.636964    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-envoy\") pod \"9d1828c6-62fc-497b-9ef9-e12d001f4683\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") "
Nov 02 15:26:44.637175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.637020    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-credential-socket\") pod \"9d1828c6-62fc-497b-9ef9-e12d001f4683\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") "
Nov 02 15:26:44.637175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.637070    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-kube-api-access-dkllj\" (UniqueName: \"kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-kube-api-access-dkllj\") pod \"9d1828c6-62fc-497b-9ef9-e12d001f4683\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") "
Nov 02 15:26:44.637175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.637083    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-data" pod "9d1828c6-62fc-497b-9ef9-e12d001f4683" (UID: "9d1828c6-62fc-497b-9ef9-e12d001f4683"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:44.637175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.637127    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-certs" pod "9d1828c6-62fc-497b-9ef9-e12d001f4683" (UID: "9d1828c6-62fc-497b-9ef9-e12d001f4683"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:44.637585 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.637204    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-podinfo\") pod \"9d1828c6-62fc-497b-9ef9-e12d001f4683\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") "
Nov 02 15:26:44.637585 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.637305    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-credential-socket" pod "9d1828c6-62fc-497b-9ef9-e12d001f4683" (UID: "9d1828c6-62fc-497b-9ef9-e12d001f4683"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:44.637585 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.637304    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-token\" (UniqueName: \"kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-token\") pod \"9d1828c6-62fc-497b-9ef9-e12d001f4683\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") "
Nov 02 15:26:44.637585 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.637418    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1828c6-62fc-497b-9ef9-e12d001f4683-istiod-ca-cert" pod "9d1828c6-62fc-497b-9ef9-e12d001f4683" (UID: "9d1828c6-62fc-497b-9ef9-e12d001f4683"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:26:44.637585 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.637426    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-socket\") pod \"9d1828c6-62fc-497b-9ef9-e12d001f4683\" (UID: \"9d1828c6-62fc-497b-9ef9-e12d001f4683\") "
Nov 02 15:26:44.637867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.637631    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-socket" pod "9d1828c6-62fc-497b-9ef9-e12d001f4683" (UID: "9d1828c6-62fc-497b-9ef9-e12d001f4683"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:44.638523 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638499    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.638624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638525    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.638624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638541    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014837c5-3da0-4c7d-a46a-92a84dac45fc-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.638624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638556    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.638624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638572    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.638624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638586    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slvmx\" (UniqueName: \"kubernetes.io/projected/014837c5-3da0-4c7d-a46a-92a84dac45fc-kube-api-access-slvmx\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.638624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638602    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.638624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638616    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.639015 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638637    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.639015 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638653    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.639015 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638667    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1828c6-62fc-497b-9ef9-e12d001f4683-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.639015 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638682    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.639015 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.638695    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014837c5-3da0-4c7d-a46a-92a84dac45fc-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.639591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.639568    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-envoy" pod "9d1828c6-62fc-497b-9ef9-e12d001f4683" (UID: "9d1828c6-62fc-497b-9ef9-e12d001f4683"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:44.639708 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.639686    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-kube-api-access-dkllj" pod "9d1828c6-62fc-497b-9ef9-e12d001f4683" (UID: "9d1828c6-62fc-497b-9ef9-e12d001f4683"). InnerVolumeSpecName "kube-api-access-dkllj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:26:44.639775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.639723    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-token" pod "9d1828c6-62fc-497b-9ef9-e12d001f4683" (UID: "9d1828c6-62fc-497b-9ef9-e12d001f4683"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:26:44.639837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.639776    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-podinfo" pod "9d1828c6-62fc-497b-9ef9-e12d001f4683" (UID: "9d1828c6-62fc-497b-9ef9-e12d001f4683"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:26:44.739799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.739764    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.739799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.739796    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkllj\" (UniqueName: \"kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-kube-api-access-dkllj\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.739799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.739807    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.740038 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.739816    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1828c6-62fc-497b-9ef9-e12d001f4683-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:26:44.938328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.929126    4891 generic.go:356] "Generic (PLEG): container finished" podID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerID="0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9" exitCode=0
Nov 02 15:26:44.938328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.929337    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" event={"ID":"9d1828c6-62fc-497b-9ef9-e12d001f4683","Type":"ContainerDied","Data":"0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9"}
Nov 02 15:26:44.938328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.929404    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm" event={"ID":"9d1828c6-62fc-497b-9ef9-e12d001f4683","Type":"ContainerDied","Data":"666e092ac947b84f2da4ea97131b86f043e5b0d647982c1dc7057733a07d1986"}
Nov 02 15:26:44.938328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.929434    4891 scope.go:122] "RemoveContainer" containerID="0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9"
Nov 02 15:26:44.938328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.929602    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"
Nov 02 15:26:44.938328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.937848    4891 generic.go:356] "Generic (PLEG): container finished" podID="014837c5-3da0-4c7d-a46a-92a84dac45fc" containerID="515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2" exitCode=0
Nov 02 15:26:44.938328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.937914    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7" event={"ID":"014837c5-3da0-4c7d-a46a-92a84dac45fc","Type":"ContainerDied","Data":"515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2"}
Nov 02 15:26:44.938328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.937961    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7" event={"ID":"014837c5-3da0-4c7d-a46a-92a84dac45fc","Type":"ContainerDied","Data":"f66c1657e84d0a5ebe7f45253a3b3acc9b18d05075833d7e48d14ec4a8c089a0"}
Nov 02 15:26:44.938328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.938035    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"
Nov 02 15:26:44.952160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.952142    4891 scope.go:122] "RemoveContainer" containerID="0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9"
Nov 02 15:26:44.952801 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:26:44.952779    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9\": container with ID starting with 0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9 not found: ID does not exist" containerID="0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9"
Nov 02 15:26:44.952896 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.952811    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9"} err="failed to get container status \"0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9\": rpc error: code = NotFound desc = could not find container \"0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9\": container with ID starting with 0c76ada56005c227770f937fa068c6ba1aaa088d3f5823f5c4968e4529f387a9 not found: ID does not exist"
Nov 02 15:26:44.952896 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.952830    4891 scope.go:122] "RemoveContainer" containerID="515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2"
Nov 02 15:26:44.962541 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.962524    4891 scope.go:122] "RemoveContainer" containerID="515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2"
Nov 02 15:26:44.962808 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:26:44.962789    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2\": container with ID starting with 515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2 not found: ID does not exist" containerID="515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2"
Nov 02 15:26:44.962867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:44.962814    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2"} err="failed to get container status \"515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2\": rpc error: code = NotFound desc = could not find container \"515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2\": container with ID starting with 515f37ce73c8aefc4af98d41ca01b9358a76ed74c59b989309bdde763802a3b2 not found: ID does not exist"
Nov 02 15:26:45.105461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:45.105427    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"]
Nov 02 15:26:45.109857 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:45.109835    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kj2h7"]
Nov 02 15:26:45.220580 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:45.220487    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"]
Nov 02 15:26:45.224609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:45.224582    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7krbm"]
Nov 02 15:26:46.524477 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:46.524438    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014837c5-3da0-4c7d-a46a-92a84dac45fc" path="/var/lib/kubelet/pods/014837c5-3da0-4c7d-a46a-92a84dac45fc/volumes"
Nov 02 15:26:46.524994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:46.524978    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" path="/var/lib/kubelet/pods/9d1828c6-62fc-497b-9ef9-e12d001f4683/volumes"
Nov 02 15:26:59.857363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.857327    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"]
Nov 02 15:26:59.859130 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859100    4891 cpu_manager.go:414] "containerMap: removing container" podUID="830e624c-98fb-48c6-a9b1-e798013a7392" containerName="revoke-keys"
Nov 02 15:26:59.859301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859285    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830e624c-98fb-48c6-a9b1-e798013a7392" containerName="revoke-keys"
Nov 02 15:26:59.859446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859431    4891 cpu_manager.go:414] "containerMap: removing container" podUID="dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" containerName="istio-proxy"
Nov 02 15:26:59.859577 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859563    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" containerName="istio-proxy"
Nov 02 15:26:59.859693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859670    4891 cpu_manager.go:414] "containerMap: removing container" podUID="d2a3b4a5-282a-473b-8d71-287a475429b1" containerName="istio-proxy"
Nov 02 15:26:59.859693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859686    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a3b4a5-282a-473b-8d71-287a475429b1" containerName="istio-proxy"
Nov 02 15:26:59.859870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859702    4891 cpu_manager.go:414] "containerMap: removing container" podUID="014837c5-3da0-4c7d-a46a-92a84dac45fc" containerName="istio-proxy"
Nov 02 15:26:59.859870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859712    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014837c5-3da0-4c7d-a46a-92a84dac45fc" containerName="istio-proxy"
Nov 02 15:26:59.859870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859725    4891 cpu_manager.go:414] "containerMap: removing container" podUID="04eab7d2-8782-4353-b7b3-1ea5df096979" containerName="istio-proxy"
Nov 02 15:26:59.859870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859732    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04eab7d2-8782-4353-b7b3-1ea5df096979" containerName="istio-proxy"
Nov 02 15:26:59.859870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859744    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy"
Nov 02 15:26:59.859870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859751    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy"
Nov 02 15:26:59.859870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859763    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1559e94a-36ed-40a4-8118-3d8f0f338a17" containerName="istio-proxy"
Nov 02 15:26:59.859870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859770    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1559e94a-36ed-40a4-8118-3d8f0f338a17" containerName="istio-proxy"
Nov 02 15:26:59.859870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859834    4891 cpu_manager.go:414] "containerMap: removing container" podUID="25b284eb-d840-4103-9ef8-7115157d7333" containerName="revoke-keys"
Nov 02 15:26:59.859870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.859841    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b284eb-d840-4103-9ef8-7115157d7333" containerName="revoke-keys"
Nov 02 15:26:59.860178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.860011    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b284eb-d840-4103-9ef8-7115157d7333" containerName="revoke-keys"
Nov 02 15:26:59.860178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.860030    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="830e624c-98fb-48c6-a9b1-e798013a7392" containerName="revoke-keys"
Nov 02 15:26:59.860178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.860038    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbc3c1fb-c245-43ca-b5f4-4ea9c08b3768" containerName="istio-proxy"
Nov 02 15:26:59.860178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.860048    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a3b4a5-282a-473b-8d71-287a475429b1" containerName="istio-proxy"
Nov 02 15:26:59.860178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.860058    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="014837c5-3da0-4c7d-a46a-92a84dac45fc" containerName="istio-proxy"
Nov 02 15:26:59.860178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.860066    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="04eab7d2-8782-4353-b7b3-1ea5df096979" containerName="istio-proxy"
Nov 02 15:26:59.860178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.860076    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1828c6-62fc-497b-9ef9-e12d001f4683" containerName="istio-proxy"
Nov 02 15:26:59.860178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.860085    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="1559e94a-36ed-40a4-8118-3d8f0f338a17" containerName="istio-proxy"
Nov 02 15:26:59.871233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.866596    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"
Nov 02 15:26:59.917261 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.917227    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"]
Nov 02 15:26:59.929577 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.929541    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcltj\" (UniqueName: \"kubernetes.io/projected/07719395-3bc0-4e9a-b84c-af507e734d1f-kube-api-access-xcltj\") pod \"payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz\" (UID: \"07719395-3bc0-4e9a-b84c-af507e734d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"
Nov 02 15:26:59.929763 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.929621    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07719395-3bc0-4e9a-b84c-af507e734d1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz\" (UID: \"07719395-3bc0-4e9a-b84c-af507e734d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"
Nov 02 15:26:59.959345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.959296    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-885c69-dockercfg-trlsw\""
Nov 02 15:26:59.959627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:26:59.959440    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-885c69\""
Nov 02 15:27:00.035332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:27:00.034568    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcltj\" (UniqueName: \"kubernetes.io/projected/07719395-3bc0-4e9a-b84c-af507e734d1f-kube-api-access-xcltj\") pod \"payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz\" (UID: \"07719395-3bc0-4e9a-b84c-af507e734d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"
Nov 02 15:27:00.035332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:27:00.034690    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07719395-3bc0-4e9a-b84c-af507e734d1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz\" (UID: \"07719395-3bc0-4e9a-b84c-af507e734d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"
Nov 02 15:27:00.039333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:27:00.035817    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07719395-3bc0-4e9a-b84c-af507e734d1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz\" (UID: \"07719395-3bc0-4e9a-b84c-af507e734d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"
Nov 02 15:27:00.161113 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:27:00.161009    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcltj\" (UniqueName: \"kubernetes.io/projected/07719395-3bc0-4e9a-b84c-af507e734d1f-kube-api-access-xcltj\") pod \"payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz\" (UID: \"07719395-3bc0-4e9a-b84c-af507e734d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"
Nov 02 15:27:00.204806 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:27:00.204773    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"
Nov 02 15:27:00.600348 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:27:00.600284    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07719395_3bc0_4e9a_b84c_af507e734d1f.slice/crio-0cbf859ed48e2e21099b3d31db6a93a23c7014a9f380de783f5b12a818849294 WatchSource:0}: Error finding container 0cbf859ed48e2e21099b3d31db6a93a23c7014a9f380de783f5b12a818849294: Status 404 returned error can't find the container with id 0cbf859ed48e2e21099b3d31db6a93a23c7014a9f380de783f5b12a818849294
Nov 02 15:27:00.600640 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:27:00.600603    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"]
Nov 02 15:27:01.100243 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:27:01.100199    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz" event={"ID":"07719395-3bc0-4e9a-b84c-af507e734d1f","Type":"ContainerStarted","Data":"bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546"}
Nov 02 15:27:01.100670 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:27:01.100263    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz" event={"ID":"07719395-3bc0-4e9a-b84c-af507e734d1f","Type":"ContainerStarted","Data":"0cbf859ed48e2e21099b3d31db6a93a23c7014a9f380de783f5b12a818849294"}
Nov 02 15:27:01.100670 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:27:01.100295    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"
Nov 02 15:27:01.220116 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:27:01.220054    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz" podStartSLOduration=2.22004188 podStartE2EDuration="2.22004188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:01.218196809 +0000 UTC m=+1828.810813611" watchObservedRunningTime="2026-11-02 15:27:01.22004188 +0000 UTC m=+1828.812658678"
Nov 02 15:27:12.111917 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:27:12.111886    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"
Nov 02 15:27:14.275779 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:27:14.275698    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:27:44.275802 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:27:44.275766    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:28:14.274616 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:28:14.274570    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:28:36.324721 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.324683    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"]
Nov 02 15:28:36.329076 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.329058    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:36.435702 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.435667    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:28:36.439789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.436064    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:28:36.439984 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.439082    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:28:36.440124 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.439714    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:28:36.442742 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.442711    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8\" (UID: \"82215bab-dd0f-4a3c-b9db-e42329343e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:36.443213 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.443192    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zm59\" (UniqueName: \"kubernetes.io/projected/82215bab-dd0f-4a3c-b9db-e42329343e34-kube-api-access-6zm59\") pod \"maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8\" (UID: \"82215bab-dd0f-4a3c-b9db-e42329343e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:36.445098 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.445077    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8\" (UID: \"82215bab-dd0f-4a3c-b9db-e42329343e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:36.462494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.462463    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"]
Nov 02 15:28:36.595260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.595168    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zm59\" (UniqueName: \"kubernetes.io/projected/82215bab-dd0f-4a3c-b9db-e42329343e34-kube-api-access-6zm59\") pod \"maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8\" (UID: \"82215bab-dd0f-4a3c-b9db-e42329343e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:36.597643 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.597616    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8\" (UID: \"82215bab-dd0f-4a3c-b9db-e42329343e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:36.598756 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.598728    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8\" (UID: \"82215bab-dd0f-4a3c-b9db-e42329343e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:36.613731 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.608760    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8\" (UID: \"82215bab-dd0f-4a3c-b9db-e42329343e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:36.629112 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.629077    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8\" (UID: \"82215bab-dd0f-4a3c-b9db-e42329343e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:36.660838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.660809    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zm59\" (UniqueName: \"kubernetes.io/projected/82215bab-dd0f-4a3c-b9db-e42329343e34-kube-api-access-6zm59\") pod \"maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8\" (UID: \"82215bab-dd0f-4a3c-b9db-e42329343e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:36.942575 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:36.942496    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:37.336446 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:28:37.336394    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82215bab_dd0f_4a3c_b9db_e42329343e34.slice/crio-3ab66698470f6ebc76d55ddca23b3d99994c9fc10b141e4872451d1f3f8c3ffe WatchSource:0}: Error finding container 3ab66698470f6ebc76d55ddca23b3d99994c9fc10b141e4872451d1f3f8c3ffe: Status 404 returned error can't find the container with id 3ab66698470f6ebc76d55ddca23b3d99994c9fc10b141e4872451d1f3f8c3ffe
Nov 02 15:28:37.378215 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:37.376558    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"]
Nov 02 15:28:37.891386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:37.891343    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8" event={"ID":"82215bab-dd0f-4a3c-b9db-e42329343e34","Type":"ContainerStarted","Data":"ebd08efbf85b008d70644dbdf1a38c0ee569a078f65c9ea5623cb9f6b6c128a7"}
Nov 02 15:28:37.891566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:37.891411    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8" event={"ID":"82215bab-dd0f-4a3c-b9db-e42329343e34","Type":"ContainerStarted","Data":"3ab66698470f6ebc76d55ddca23b3d99994c9fc10b141e4872451d1f3f8c3ffe"}
Nov 02 15:28:38.009555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:38.009453    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8" podStartSLOduration=2.009440287 podStartE2EDuration="2.009440287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:38.005999262 +0000 UTC m=+1925.598616062" watchObservedRunningTime="2026-11-02 15:28:38.009440287 +0000 UTC m=+1925.602057087"
Nov 02 15:28:38.900108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:38.900065    4891 generic.go:356] "Generic (PLEG): container finished" podID="82215bab-dd0f-4a3c-b9db-e42329343e34" containerID="ebd08efbf85b008d70644dbdf1a38c0ee569a078f65c9ea5623cb9f6b6c128a7" exitCode=0
Nov 02 15:28:38.900623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:38.900156    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8" event={"ID":"82215bab-dd0f-4a3c-b9db-e42329343e34","Type":"ContainerDied","Data":"ebd08efbf85b008d70644dbdf1a38c0ee569a078f65c9ea5623cb9f6b6c128a7"}
Nov 02 15:28:40.099357 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.099294    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:40.165979 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.165876    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-xks-ca\") pod \"82215bab-dd0f-4a3c-b9db-e42329343e34\" (UID: \"82215bab-dd0f-4a3c-b9db-e42329343e34\") "
Nov 02 15:28:40.166259 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.166234    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82215bab-dd0f-4a3c-b9db-e42329343e34-kube-api-access-6zm59\" (UniqueName: \"kubernetes.io/projected/82215bab-dd0f-4a3c-b9db-e42329343e34-kube-api-access-6zm59\") pod \"82215bab-dd0f-4a3c-b9db-e42329343e34\" (UID: \"82215bab-dd0f-4a3c-b9db-e42329343e34\") "
Nov 02 15:28:40.166462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.166445    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-openshift-ca\") pod \"82215bab-dd0f-4a3c-b9db-e42329343e34\" (UID: \"82215bab-dd0f-4a3c-b9db-e42329343e34\") "
Nov 02 15:28:40.178621 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.178582    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-openshift-ca" pod "82215bab-dd0f-4a3c-b9db-e42329343e34" (UID: "82215bab-dd0f-4a3c-b9db-e42329343e34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:28:40.196369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.196297    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-xks-ca" pod "82215bab-dd0f-4a3c-b9db-e42329343e34" (UID: "82215bab-dd0f-4a3c-b9db-e42329343e34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:28:40.207741 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.207679    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82215bab-dd0f-4a3c-b9db-e42329343e34-kube-api-access-6zm59" pod "82215bab-dd0f-4a3c-b9db-e42329343e34" (UID: "82215bab-dd0f-4a3c-b9db-e42329343e34"). InnerVolumeSpecName "kube-api-access-6zm59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:40.270706 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.270619    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:28:40.270706 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.270662    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zm59\" (UniqueName: \"kubernetes.io/projected/82215bab-dd0f-4a3c-b9db-e42329343e34-kube-api-access-6zm59\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:28:40.270706 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.270679    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82215bab-dd0f-4a3c-b9db-e42329343e34-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:28:40.920250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.920169    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"
Nov 02 15:28:40.920250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.920207    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8" event={"ID":"82215bab-dd0f-4a3c-b9db-e42329343e34","Type":"ContainerDied","Data":"3ab66698470f6ebc76d55ddca23b3d99994c9fc10b141e4872451d1f3f8c3ffe"}
Nov 02 15:28:40.920473 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:28:40.920265    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ab66698470f6ebc76d55ddca23b3d99994c9fc10b141e4872451d1f3f8c3ffe"
Nov 02 15:28:44.275451 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:28:44.275409    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:29:14.274924 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:14.274885    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:29:15.763947 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.763912    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"]
Nov 02 15:29:15.764613 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.764593    4891 cpu_manager.go:414] "containerMap: removing container" podUID="82215bab-dd0f-4a3c-b9db-e42329343e34" containerName="revoke-keys"
Nov 02 15:29:15.764613 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.764612    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82215bab-dd0f-4a3c-b9db-e42329343e34" containerName="revoke-keys"
Nov 02 15:29:15.764745 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.764736    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="82215bab-dd0f-4a3c-b9db-e42329343e34" containerName="revoke-keys"
Nov 02 15:29:15.768480 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.768459    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:15.796609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.796574    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-53fd96-49js8\" (UID: \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:15.796609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.796610    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-53fd96-49js8\" (UID: \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:15.796795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.796764    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk5qf\" (UniqueName: \"kubernetes.io/projected/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-kube-api-access-qk5qf\") pod \"maas-api-revoke-keys-e2e-api-a-53fd96-49js8\" (UID: \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:15.820632 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.820597    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:29:15.870133 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.870100    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:29:15.870421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.870125    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:29:15.870587 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.870139    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:29:15.877567 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.877537    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"]
Nov 02 15:29:15.897782 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.897745    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-53fd96-49js8\" (UID: \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:15.897902 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.897787    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-53fd96-49js8\" (UID: \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:15.897951 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.897897    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk5qf\" (UniqueName: \"kubernetes.io/projected/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-kube-api-access-qk5qf\") pod \"maas-api-revoke-keys-e2e-api-a-53fd96-49js8\" (UID: \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:15.898608 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.898582    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-53fd96-49js8\" (UID: \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:15.900231 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.900210    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-53fd96-49js8\" (UID: \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:15.958910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:15.958873    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk5qf\" (UniqueName: \"kubernetes.io/projected/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-kube-api-access-qk5qf\") pod \"maas-api-revoke-keys-e2e-api-a-53fd96-49js8\" (UID: \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:16.082293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:16.082256    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:16.474296 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:16.474266    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"]
Nov 02 15:29:16.474453 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:29:16.474412    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f15ea9_ca87_40d6_941b_7d4f4e575a18.slice/crio-483e73d51622ea334d802831059c76a4259a9b44fd970931a4be7123cbbce2f9 WatchSource:0}: Error finding container 483e73d51622ea334d802831059c76a4259a9b44fd970931a4be7123cbbce2f9: Status 404 returned error can't find the container with id 483e73d51622ea334d802831059c76a4259a9b44fd970931a4be7123cbbce2f9
Nov 02 15:29:17.216642 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:17.216555    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8" event={"ID":"d3f15ea9-ca87-40d6-941b-7d4f4e575a18","Type":"ContainerStarted","Data":"443b33cdae15e6aaa0a8b1495ea37cbf456b95384cb4927f77423d7e1a9156c0"}
Nov 02 15:29:17.216642 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:17.216608    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8" event={"ID":"d3f15ea9-ca87-40d6-941b-7d4f4e575a18","Type":"ContainerStarted","Data":"483e73d51622ea334d802831059c76a4259a9b44fd970931a4be7123cbbce2f9"}
Nov 02 15:29:17.386239 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:17.386183    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8" podStartSLOduration=2.386169445 podStartE2EDuration="2.386169445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:17.333028287 +0000 UTC m=+1964.925645090" watchObservedRunningTime="2026-11-02 15:29:17.386169445 +0000 UTC m=+1964.978786290"
Nov 02 15:29:18.225851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:18.225810    4891 generic.go:356] "Generic (PLEG): container finished" podID="d3f15ea9-ca87-40d6-941b-7d4f4e575a18" containerID="443b33cdae15e6aaa0a8b1495ea37cbf456b95384cb4927f77423d7e1a9156c0" exitCode=0
Nov 02 15:29:18.226354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:18.225892    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8" event={"ID":"d3f15ea9-ca87-40d6-941b-7d4f4e575a18","Type":"ContainerDied","Data":"443b33cdae15e6aaa0a8b1495ea37cbf456b95384cb4927f77423d7e1a9156c0"}
Nov 02 15:29:19.398264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:19.398236    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:19.450597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:19.450561    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-openshift-ca\") pod \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\" (UID: \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\") "
Nov 02 15:29:19.450820 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:19.450609    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-kube-api-access-qk5qf\" (UniqueName: \"kubernetes.io/projected/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-kube-api-access-qk5qf\") pod \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\" (UID: \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\") "
Nov 02 15:29:19.450820 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:19.450776    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-xks-ca\") pod \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\" (UID: \"d3f15ea9-ca87-40d6-941b-7d4f4e575a18\") "
Nov 02 15:29:19.451006 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:19.450975    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-openshift-ca" pod "d3f15ea9-ca87-40d6-941b-7d4f4e575a18" (UID: "d3f15ea9-ca87-40d6-941b-7d4f4e575a18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:19.451483 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:19.451458    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:19.453362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:19.453300    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-kube-api-access-qk5qf" pod "d3f15ea9-ca87-40d6-941b-7d4f4e575a18" (UID: "d3f15ea9-ca87-40d6-941b-7d4f4e575a18"). InnerVolumeSpecName "kube-api-access-qk5qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:19.453362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:19.453296    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-xks-ca" pod "d3f15ea9-ca87-40d6-941b-7d4f4e575a18" (UID: "d3f15ea9-ca87-40d6-941b-7d4f4e575a18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:29:19.552304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:19.552274    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk5qf\" (UniqueName: \"kubernetes.io/projected/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-kube-api-access-qk5qf\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:19.552561 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:19.552306    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f15ea9-ca87-40d6-941b-7d4f4e575a18-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:20.243396 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:20.243358    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8" event={"ID":"d3f15ea9-ca87-40d6-941b-7d4f4e575a18","Type":"ContainerDied","Data":"483e73d51622ea334d802831059c76a4259a9b44fd970931a4be7123cbbce2f9"}
Nov 02 15:29:20.243574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:20.243422    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="483e73d51622ea334d802831059c76a4259a9b44fd970931a4be7123cbbce2f9"
Nov 02 15:29:20.243574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:20.243369    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"
Nov 02 15:29:25.248160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:25.248120    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"]
Nov 02 15:29:25.248660 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:25.248427    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerName="istio-proxy" containerID="cri-o://4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1" gracePeriod=30
Nov 02 15:29:25.300917 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:25.300881    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"]
Nov 02 15:29:25.301162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:25.301133    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh" podUID="40993be1-454f-4d83-8b5a-85c1975f71da" containerName="istio-proxy" containerID="cri-o://ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a" gracePeriod=30
Nov 02 15:29:25.307894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:25.307866    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"]
Nov 02 15:29:25.308162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:25.308101    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy" containerID="cri-o://a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b" gracePeriod=30
Nov 02 15:29:25.379625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:25.379581    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"]
Nov 02 15:29:25.379837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:25.379809    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" podUID="15bfe010-d72c-42f1-918c-8ec54254da96" containerName="istio-proxy" containerID="cri-o://e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9" gracePeriod=30
Nov 02 15:29:25.385053 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:25.385022    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"]
Nov 02 15:29:25.385238 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:25.385216    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerName="istio-proxy" containerID="cri-o://d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256" gracePeriod=30
Nov 02 15:29:25.429668 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:25.429632    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"]
Nov 02 15:29:25.429892 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:25.429868    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" podUID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" containerName="istio-proxy" containerID="cri-o://c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3" gracePeriod=30
Nov 02 15:29:27.363609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:27.363557    4891 patch_prober.go:29] interesting pod/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:29:27.364130 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:27.363638    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:29:27.925197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:27.925162    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:29:27.925423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:27.925232    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" podUID="15bfe010-d72c-42f1-918c-8ec54254da96" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:29:28.354959 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:28.354922    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"]
Nov 02 15:29:28.355176 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:28.355135    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerName="istio-proxy" containerID="cri-o://f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50" gracePeriod=30
Nov 02 15:29:28.692392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:28.692282    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:29:28.692779 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:28.692382    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" podUID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:29:30.052355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.052300    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"]
Nov 02 15:29:30.101466 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.101433    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16f0068-8ks7w"]
Nov 02 15:29:30.327762 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.327690    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:29:30.327895 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.327762    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:29:30.595931 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.595848    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830e624c-98fb-48c6-a9b1-e798013a7392" path="/var/lib/kubelet/pods/830e624c-98fb-48c6-a9b1-e798013a7392/volumes"
Nov 02 15:29:30.791584 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.791552    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:29:30.862745 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.862717    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:29:30.869526 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.869500    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:29:30.869665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.869601    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-data\") pod \"806a03af-da93-4064-a70f-cc3ee68e5c9d\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") "
Nov 02 15:29:30.869738 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.869721    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-kube-api-access-f9fnx\" (UniqueName: \"kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-kube-api-access-f9fnx\") pod \"806a03af-da93-4064-a70f-cc3ee68e5c9d\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") "
Nov 02 15:29:30.869794 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.869768    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806a03af-da93-4064-a70f-cc3ee68e5c9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806a03af-da93-4064-a70f-cc3ee68e5c9d-istiod-ca-cert\") pod \"806a03af-da93-4064-a70f-cc3ee68e5c9d\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") "
Nov 02 15:29:30.869895 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.869880    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-certs\") pod \"806a03af-da93-4064-a70f-cc3ee68e5c9d\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") "
Nov 02 15:29:30.869951 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.869910    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-envoy\") pod \"806a03af-da93-4064-a70f-cc3ee68e5c9d\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") "
Nov 02 15:29:30.870300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.870061    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-podinfo\") pod \"806a03af-da93-4064-a70f-cc3ee68e5c9d\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") "
Nov 02 15:29:30.870300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.870100    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-socket\") pod \"806a03af-da93-4064-a70f-cc3ee68e5c9d\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") "
Nov 02 15:29:30.870300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.870146    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-credential-socket\") pod \"806a03af-da93-4064-a70f-cc3ee68e5c9d\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") "
Nov 02 15:29:30.870300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.870235    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-token\" (UniqueName: \"kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-token\") pod \"806a03af-da93-4064-a70f-cc3ee68e5c9d\" (UID: \"806a03af-da93-4064-a70f-cc3ee68e5c9d\") "
Nov 02 15:29:30.870300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.870237    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-certs" pod "806a03af-da93-4064-a70f-cc3ee68e5c9d" (UID: "806a03af-da93-4064-a70f-cc3ee68e5c9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.870300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.870252    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-data" pod "806a03af-da93-4064-a70f-cc3ee68e5c9d" (UID: "806a03af-da93-4064-a70f-cc3ee68e5c9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.870833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.870805    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-socket" pod "806a03af-da93-4064-a70f-cc3ee68e5c9d" (UID: "806a03af-da93-4064-a70f-cc3ee68e5c9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.870910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.870867    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806a03af-da93-4064-a70f-cc3ee68e5c9d-istiod-ca-cert" pod "806a03af-da93-4064-a70f-cc3ee68e5c9d" (UID: "806a03af-da93-4064-a70f-cc3ee68e5c9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:30.870910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.870894    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.871040 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.870913    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.871092 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.871060    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-credential-socket" pod "806a03af-da93-4064-a70f-cc3ee68e5c9d" (UID: "806a03af-da93-4064-a70f-cc3ee68e5c9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.874630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.874603    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-token" pod "806a03af-da93-4064-a70f-cc3ee68e5c9d" (UID: "806a03af-da93-4064-a70f-cc3ee68e5c9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:30.875903 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.875869    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-podinfo" pod "806a03af-da93-4064-a70f-cc3ee68e5c9d" (UID: "806a03af-da93-4064-a70f-cc3ee68e5c9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:29:30.877586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.877560    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-kube-api-access-f9fnx" pod "806a03af-da93-4064-a70f-cc3ee68e5c9d" (UID: "806a03af-da93-4064-a70f-cc3ee68e5c9d"). InnerVolumeSpecName "kube-api-access-f9fnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:30.879715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.879681    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-envoy" pod "806a03af-da93-4064-a70f-cc3ee68e5c9d" (UID: "806a03af-da93-4064-a70f-cc3ee68e5c9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.916732 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.914833    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:29:30.917574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.917244    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:29:30.926065 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.926039    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:29:30.972199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972165    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15bfe010-d72c-42f1-918c-8ec54254da96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bfe010-d72c-42f1-918c-8ec54254da96-istio-podinfo\") pod \"15bfe010-d72c-42f1-918c-8ec54254da96\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") "
Nov 02 15:29:30.972434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972218    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-certs\") pod \"5ee0893d-8caa-4d67-971c-d51bd8955783\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") "
Nov 02 15:29:30.972434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972268    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-credential-socket\") pod \"15bfe010-d72c-42f1-918c-8ec54254da96\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") "
Nov 02 15:29:30.972434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972295    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-data\") pod \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") "
Nov 02 15:29:30.972434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972333    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-envoy\") pod \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") "
Nov 02 15:29:30.972434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972363    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-envoy\") pod \"40993be1-454f-4d83-8b5a-85c1975f71da\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") "
Nov 02 15:29:30.972434 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972406    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istiod-ca-cert\") pod \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") "
Nov 02 15:29:30.972768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972449    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40993be1-454f-4d83-8b5a-85c1975f71da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40993be1-454f-4d83-8b5a-85c1975f71da-istiod-ca-cert\") pod \"40993be1-454f-4d83-8b5a-85c1975f71da\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") "
Nov 02 15:29:30.972768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972514    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-certs\") pod \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") "
Nov 02 15:29:30.972768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972556    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-certs\") pod \"15bfe010-d72c-42f1-918c-8ec54254da96\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") "
Nov 02 15:29:30.972768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972586    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istiod-ca-cert\") pod \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") "
Nov 02 15:29:30.972768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972623    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-socket\") pod \"5ee0893d-8caa-4d67-971c-d51bd8955783\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") "
Nov 02 15:29:30.972768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972675    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-certs\") pod \"40993be1-454f-4d83-8b5a-85c1975f71da\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") "
Nov 02 15:29:30.972768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972677    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-data" pod "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" (UID: "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.972768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972705    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-envoy\") pod \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") "
Nov 02 15:29:30.972768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972737    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-istio-token\" (UniqueName: \"kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-istio-token\") pod \"15bfe010-d72c-42f1-918c-8ec54254da96\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") "
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972795    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-podinfo\") pod \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") "
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972831    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-data\") pod \"15bfe010-d72c-42f1-918c-8ec54254da96\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") "
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972865    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40993be1-454f-4d83-8b5a-85c1975f71da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40993be1-454f-4d83-8b5a-85c1975f71da-istio-podinfo\") pod \"40993be1-454f-4d83-8b5a-85c1975f71da\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") "
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972873    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-certs" pod "5ee0893d-8caa-4d67-971c-d51bd8955783" (UID: "5ee0893d-8caa-4d67-971c-d51bd8955783"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972900    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-kube-api-access-kqt6c\" (UniqueName: \"kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-kube-api-access-kqt6c\") pod \"15bfe010-d72c-42f1-918c-8ec54254da96\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") "
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972946    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-kube-api-access-h6k9c\" (UniqueName: \"kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-kube-api-access-h6k9c\") pod \"40993be1-454f-4d83-8b5a-85c1975f71da\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") "
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.972984    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-token\" (UniqueName: \"kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-token\") pod \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") "
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.973014    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-credential-socket\") pod \"5ee0893d-8caa-4d67-971c-d51bd8955783\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") "
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.973040    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-credential-socket" pod "15bfe010-d72c-42f1-918c-8ec54254da96" (UID: "15bfe010-d72c-42f1-918c-8ec54254da96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.973050    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-envoy\") pod \"5ee0893d-8caa-4d67-971c-d51bd8955783\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") "
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.973083    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-data\") pod \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") "
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.973114    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-kube-api-access-7xnh6\" (UniqueName: \"kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-kube-api-access-7xnh6\") pod \"5ee0893d-8caa-4d67-971c-d51bd8955783\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") "
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.973172    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-certs" pod "40993be1-454f-4d83-8b5a-85c1975f71da" (UID: "40993be1-454f-4d83-8b5a-85c1975f71da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.973291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.973171    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-kube-api-access-wpltz\" (UniqueName: \"kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-kube-api-access-wpltz\") pod \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") "
Nov 02 15:29:30.974068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.973771    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-certs" pod "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" (UID: "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.974068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.973900    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istiod-ca-cert" pod "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" (UID: "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:30.974187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.974129    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40993be1-454f-4d83-8b5a-85c1975f71da-istiod-ca-cert" pod "40993be1-454f-4d83-8b5a-85c1975f71da" (UID: "40993be1-454f-4d83-8b5a-85c1975f71da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.974417    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-credential-socket" pod "5ee0893d-8caa-4d67-971c-d51bd8955783" (UID: "5ee0893d-8caa-4d67-971c-d51bd8955783"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.974929    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istiod-ca-cert" pod "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" (UID: "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.975283    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-certs" pod "15bfe010-d72c-42f1-918c-8ec54254da96" (UID: "15bfe010-d72c-42f1-918c-8ec54254da96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976491    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-podinfo\") pod \"5ee0893d-8caa-4d67-971c-d51bd8955783\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976521    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee0893d-8caa-4d67-971c-d51bd8955783-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee0893d-8caa-4d67-971c-d51bd8955783-istiod-ca-cert\") pod \"5ee0893d-8caa-4d67-971c-d51bd8955783\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976555    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-socket\") pod \"40993be1-454f-4d83-8b5a-85c1975f71da\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976619    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-credential-socket\") pod \"40993be1-454f-4d83-8b5a-85c1975f71da\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976640    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-kube-api-access-2b5sj\" (UniqueName: \"kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-kube-api-access-2b5sj\") pod \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976659    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-certs\") pod \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976670    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-socket" pod "5ee0893d-8caa-4d67-971c-d51bd8955783" (UID: "5ee0893d-8caa-4d67-971c-d51bd8955783"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976722    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-podinfo\") pod \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976740    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-istio-token\" (UniqueName: \"kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-istio-token\") pod \"40993be1-454f-4d83-8b5a-85c1975f71da\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976783    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-credential-socket\") pod \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976815    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-token\" (UniqueName: \"kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-token\") pod \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976868    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-socket\") pod \"15bfe010-d72c-42f1-918c-8ec54254da96\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976951    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-envoy\") pod \"15bfe010-d72c-42f1-918c-8ec54254da96\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976980    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15bfe010-d72c-42f1-918c-8ec54254da96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bfe010-d72c-42f1-918c-8ec54254da96-istiod-ca-cert\") pod \"15bfe010-d72c-42f1-918c-8ec54254da96\" (UID: \"15bfe010-d72c-42f1-918c-8ec54254da96\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.977019    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-token\" (UniqueName: \"kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-token\") pod \"5ee0893d-8caa-4d67-971c-d51bd8955783\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.977051    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-socket\") pod \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.977098    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-data\") pod \"40993be1-454f-4d83-8b5a-85c1975f71da\" (UID: \"40993be1-454f-4d83-8b5a-85c1975f71da\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.977148    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-credential-socket\") pod \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\" (UID: \"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.977184    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-socket\") pod \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\" (UID: \"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.977229    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-data\") pod \"5ee0893d-8caa-4d67-971c-d51bd8955783\" (UID: \"5ee0893d-8caa-4d67-971c-d51bd8955783\") "
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978160    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978189    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978205    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978224    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978239    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978255    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978269    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978285    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9fnx\" (UniqueName: \"kubernetes.io/projected/806a03af-da93-4064-a70f-cc3ee68e5c9d-kube-api-access-f9fnx\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978303    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806a03af-da93-4064-a70f-cc3ee68e5c9d-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978334    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978353    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978368    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978383    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978396    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806a03af-da93-4064-a70f-cc3ee68e5c9d-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978410    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978429    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978449    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40993be1-454f-4d83-8b5a-85c1975f71da-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.979372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978463    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806a03af-da93-4064-a70f-cc3ee68e5c9d-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:30.981616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.976813    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-socket" pod "40993be1-454f-4d83-8b5a-85c1975f71da" (UID: "40993be1-454f-4d83-8b5a-85c1975f71da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.981616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.977041    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-data" pod "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" (UID: "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.981616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.977243    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-credential-socket" pod "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" (UID: "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.981616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.977327    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee0893d-8caa-4d67-971c-d51bd8955783-istiod-ca-cert" pod "5ee0893d-8caa-4d67-971c-d51bd8955783" (UID: "5ee0893d-8caa-4d67-971c-d51bd8955783"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:30.981616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.977452    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-credential-socket" pod "40993be1-454f-4d83-8b5a-85c1975f71da" (UID: "40993be1-454f-4d83-8b5a-85c1975f71da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.981616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.977588    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-data" pod "15bfe010-d72c-42f1-918c-8ec54254da96" (UID: "15bfe010-d72c-42f1-918c-8ec54254da96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.981616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.977890    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15bfe010-d72c-42f1-918c-8ec54254da96-istiod-ca-cert" pod "15bfe010-d72c-42f1-918c-8ec54254da96" (UID: "15bfe010-d72c-42f1-918c-8ec54254da96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:30.981616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.978671    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-data" pod "5ee0893d-8caa-4d67-971c-d51bd8955783" (UID: "5ee0893d-8caa-4d67-971c-d51bd8955783"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.981616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.980903    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-certs" pod "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" (UID: "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.981616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.981075    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-data" pod "40993be1-454f-4d83-8b5a-85c1975f71da" (UID: "40993be1-454f-4d83-8b5a-85c1975f71da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.981616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.981277    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-socket" pod "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" (UID: "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.981616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.981386    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-socket" pod "15bfe010-d72c-42f1-918c-8ec54254da96" (UID: "15bfe010-d72c-42f1-918c-8ec54254da96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.982193 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.981646    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-credential-socket" pod "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" (UID: "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.982193 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.981881    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-socket" pod "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" (UID: "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.983291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.983103    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-kube-api-access-h6k9c" pod "40993be1-454f-4d83-8b5a-85c1975f71da" (UID: "40993be1-454f-4d83-8b5a-85c1975f71da"). InnerVolumeSpecName "kube-api-access-h6k9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:30.983291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.983171    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-kube-api-access-wpltz" pod "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" (UID: "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff"). InnerVolumeSpecName "kube-api-access-wpltz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:30.983291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.983170    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-token" pod "5ee0893d-8caa-4d67-971c-d51bd8955783" (UID: "5ee0893d-8caa-4d67-971c-d51bd8955783"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:30.983291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.983246    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-envoy" pod "40993be1-454f-4d83-8b5a-85c1975f71da" (UID: "40993be1-454f-4d83-8b5a-85c1975f71da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.983612 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.983347    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15bfe010-d72c-42f1-918c-8ec54254da96-istio-podinfo" pod "15bfe010-d72c-42f1-918c-8ec54254da96" (UID: "15bfe010-d72c-42f1-918c-8ec54254da96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:29:30.983677 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.983627    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-envoy" pod "5ee0893d-8caa-4d67-971c-d51bd8955783" (UID: "5ee0893d-8caa-4d67-971c-d51bd8955783"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.983677 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.983629    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-envoy" pod "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" (UID: "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.983784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.983677    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-token" pod "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" (UID: "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:30.983784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.983691    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-token" pod "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" (UID: "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:30.983784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.983747    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-istio-token" pod "15bfe010-d72c-42f1-918c-8ec54254da96" (UID: "15bfe010-d72c-42f1-918c-8ec54254da96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:30.984051 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.984018    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-envoy" pod "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" (UID: "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.984222 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.984200    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-kube-api-access-kqt6c" pod "15bfe010-d72c-42f1-918c-8ec54254da96" (UID: "15bfe010-d72c-42f1-918c-8ec54254da96"). InnerVolumeSpecName "kube-api-access-kqt6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:30.984709 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.984690    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-kube-api-access-2b5sj" pod "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" (UID: "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5"). InnerVolumeSpecName "kube-api-access-2b5sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:30.986661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.986639    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-podinfo" pod "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" (UID: "4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:29:30.986864 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.986851    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-kube-api-access-7xnh6" pod "5ee0893d-8caa-4d67-971c-d51bd8955783" (UID: "5ee0893d-8caa-4d67-971c-d51bd8955783"). InnerVolumeSpecName "kube-api-access-7xnh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:30.988015 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.987995    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-istio-token" pod "40993be1-454f-4d83-8b5a-85c1975f71da" (UID: "40993be1-454f-4d83-8b5a-85c1975f71da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:30.988280 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.988258    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-envoy" pod "15bfe010-d72c-42f1-918c-8ec54254da96" (UID: "15bfe010-d72c-42f1-918c-8ec54254da96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:30.988377 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.988341    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40993be1-454f-4d83-8b5a-85c1975f71da-istio-podinfo" pod "40993be1-454f-4d83-8b5a-85c1975f71da" (UID: "40993be1-454f-4d83-8b5a-85c1975f71da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:29:30.988377 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.988356    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-podinfo" pod "5ee0893d-8caa-4d67-971c-d51bd8955783" (UID: "5ee0893d-8caa-4d67-971c-d51bd8955783"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:29:30.988451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:30.988400    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-podinfo" pod "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" (UID: "666c6a6f-4079-4d31-b6ff-1f2c1116a0ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:29:31.080452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080408    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080444    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080462    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bfe010-d72c-42f1-918c-8ec54254da96-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080475    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080491    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080505    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080519    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080533    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080549    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080562    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bfe010-d72c-42f1-918c-8ec54254da96-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080576    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080591    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080604    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080617    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080633    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080646    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bfe010-d72c-42f1-918c-8ec54254da96-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080660    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40993be1-454f-4d83-8b5a-85c1975f71da-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080673    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqt6c\" (UniqueName: \"kubernetes.io/projected/15bfe010-d72c-42f1-918c-8ec54254da96-kube-api-access-kqt6c\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080690    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6k9c\" (UniqueName: \"kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-kube-api-access-h6k9c\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080704    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080719    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080733    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080749    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xnh6\" (UniqueName: \"kubernetes.io/projected/5ee0893d-8caa-4d67-971c-d51bd8955783-kube-api-access-7xnh6\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080763    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpltz\" (UniqueName: \"kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-kube-api-access-wpltz\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080776    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee0893d-8caa-4d67-971c-d51bd8955783-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080790    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee0893d-8caa-4d67-971c-d51bd8955783-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080804    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080818    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40993be1-454f-4d83-8b5a-85c1975f71da-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080832    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b5sj\" (UniqueName: \"kubernetes.io/projected/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-kube-api-access-2b5sj\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080846    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080859    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080873    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40993be1-454f-4d83-8b5a-85c1975f71da-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080886    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.080911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.080900    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:31.335680 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.335636    4891 generic.go:356] "Generic (PLEG): container finished" podID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerID="4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1" exitCode=0
Nov 02 15:29:31.335881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.335694    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"
Nov 02 15:29:31.335881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.335719    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" event={"ID":"806a03af-da93-4064-a70f-cc3ee68e5c9d","Type":"ContainerDied","Data":"4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1"}
Nov 02 15:29:31.335881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.335784    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679" event={"ID":"806a03af-da93-4064-a70f-cc3ee68e5c9d","Type":"ContainerDied","Data":"92ed873a603437829b717ee2093e7e5bf1199d04c0f519ccf9cce679336e0992"}
Nov 02 15:29:31.335881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.335815    4891 scope.go:122] "RemoveContainer" containerID="4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1"
Nov 02 15:29:31.337824 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.337795    4891 generic.go:356] "Generic (PLEG): container finished" podID="15bfe010-d72c-42f1-918c-8ec54254da96" containerID="e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9" exitCode=0
Nov 02 15:29:31.337944 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.337910    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" event={"ID":"15bfe010-d72c-42f1-918c-8ec54254da96","Type":"ContainerDied","Data":"e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9"}
Nov 02 15:29:31.338019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.337956    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws" event={"ID":"15bfe010-d72c-42f1-918c-8ec54254da96","Type":"ContainerDied","Data":"05ed84586a91282985e76916c4d6950680f1d6183d97bd8b8041a9d99567850c"}
Nov 02 15:29:31.338074 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.338030    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"
Nov 02 15:29:31.348770 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.348741    4891 generic.go:356] "Generic (PLEG): container finished" podID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerID="d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256" exitCode=0
Nov 02 15:29:31.348915 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.348904    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"
Nov 02 15:29:31.350504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.350129    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" event={"ID":"5ee0893d-8caa-4d67-971c-d51bd8955783","Type":"ContainerDied","Data":"d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256"}
Nov 02 15:29:31.350504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.350177    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg" event={"ID":"5ee0893d-8caa-4d67-971c-d51bd8955783","Type":"ContainerDied","Data":"7351cfad643e9bf821221a26e07bc36a122828395c073b3ae8cbb34f4359389b"}
Nov 02 15:29:31.352356 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.352226    4891 generic.go:356] "Generic (PLEG): container finished" podID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerID="a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b" exitCode=0
Nov 02 15:29:31.352490 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.352466    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" event={"ID":"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff","Type":"ContainerDied","Data":"a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b"}
Nov 02 15:29:31.352579 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.352521    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk" event={"ID":"666c6a6f-4079-4d31-b6ff-1f2c1116a0ff","Type":"ContainerDied","Data":"4fa549674da7a9fa2c0d9773cd520ea331e7bf9b2f53bb04f02b664c356aa81a"}
Nov 02 15:29:31.352647 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.352577    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"
Nov 02 15:29:31.353048 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.353023    4891 scope.go:122] "RemoveContainer" containerID="4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1"
Nov 02 15:29:31.353625 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:31.353597    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1\": container with ID starting with 4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1 not found: ID does not exist" containerID="4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1"
Nov 02 15:29:31.353701 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.353637    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1"} err="failed to get container status \"4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1\": rpc error: code = NotFound desc = could not find container \"4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1\": container with ID starting with 4c4d8ee693e79b90c474e6661ffd2333757d0e0ac5b4d6d45da9aed6a89900c1 not found: ID does not exist"
Nov 02 15:29:31.353701 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.353662    4891 scope.go:122] "RemoveContainer" containerID="e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9"
Nov 02 15:29:31.354610 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.354548    4891 generic.go:356] "Generic (PLEG): container finished" podID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" containerID="c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3" exitCode=0
Nov 02 15:29:31.354696 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.354677    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" event={"ID":"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5","Type":"ContainerDied","Data":"c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3"}
Nov 02 15:29:31.354746 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.354718    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl" event={"ID":"4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5","Type":"ContainerDied","Data":"5c1b00a23bee43100c215037550048de77b4fa8694163894320f1c6f28ffc882"}
Nov 02 15:29:31.354785 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.354779    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"
Nov 02 15:29:31.363129 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.363098    4891 generic.go:356] "Generic (PLEG): container finished" podID="40993be1-454f-4d83-8b5a-85c1975f71da" containerID="ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a" exitCode=0
Nov 02 15:29:31.363243 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.363183    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh" event={"ID":"40993be1-454f-4d83-8b5a-85c1975f71da","Type":"ContainerDied","Data":"ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a"}
Nov 02 15:29:31.363243 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.363226    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh" event={"ID":"40993be1-454f-4d83-8b5a-85c1975f71da","Type":"ContainerDied","Data":"ee31be4ee17016bb1d11a9cf93ba18f02748ea834a219b643863767cd02175d9"}
Nov 02 15:29:31.363392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.363296    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"
Nov 02 15:29:31.369837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.369816    4891 scope.go:122] "RemoveContainer" containerID="e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9"
Nov 02 15:29:31.370726 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:31.370558    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9\": container with ID starting with e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9 not found: ID does not exist" containerID="e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9"
Nov 02 15:29:31.370726 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.370611    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9"} err="failed to get container status \"e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9\": rpc error: code = NotFound desc = could not find container \"e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9\": container with ID starting with e29185830a525621d99c3ee60c9ed137bd0ec5bfa4c685eefe44684eef026ca9 not found: ID does not exist"
Nov 02 15:29:31.370726 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.370634    4891 scope.go:122] "RemoveContainer" containerID="d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256"
Nov 02 15:29:31.402577 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.402541    4891 scope.go:122] "RemoveContainer" containerID="d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256"
Nov 02 15:29:31.403167 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:31.403130    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256\": container with ID starting with d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256 not found: ID does not exist" containerID="d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256"
Nov 02 15:29:31.403293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.403176    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256"} err="failed to get container status \"d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256\": rpc error: code = NotFound desc = could not find container \"d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256\": container with ID starting with d12f63ad0dd94e4bca70825bb75c719ab2161a87ec52dfbbef336ac7851cd256 not found: ID does not exist"
Nov 02 15:29:31.403293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.403203    4891 scope.go:122] "RemoveContainer" containerID="a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b"
Nov 02 15:29:31.418960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.418923    4891 scope.go:122] "RemoveContainer" containerID="a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b"
Nov 02 15:29:31.419685 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:31.419377    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b\": container with ID starting with a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b not found: ID does not exist" containerID="a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b"
Nov 02 15:29:31.419685 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.419417    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b"} err="failed to get container status \"a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b\": rpc error: code = NotFound desc = could not find container \"a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b\": container with ID starting with a2d7bed9838f5fe39d86035e72d92b0beec009a6b1ad7883b9ff23261bb7160b not found: ID does not exist"
Nov 02 15:29:31.419685 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.419450    4891 scope.go:122] "RemoveContainer" containerID="c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3"
Nov 02 15:29:31.430779 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.430756    4891 scope.go:122] "RemoveContainer" containerID="c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3"
Nov 02 15:29:31.431094 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:31.431061    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3\": container with ID starting with c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3 not found: ID does not exist" containerID="c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3"
Nov 02 15:29:31.431189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.431105    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3"} err="failed to get container status \"c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3\": rpc error: code = NotFound desc = could not find container \"c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3\": container with ID starting with c3f132a56d83feea7c52c4b4267f485b4de2c11142e97ec8258cf0063dcc7ed3 not found: ID does not exist"
Nov 02 15:29:31.431189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.431131    4891 scope.go:122] "RemoveContainer" containerID="ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a"
Nov 02 15:29:31.442473 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.442440    4891 scope.go:122] "RemoveContainer" containerID="ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a"
Nov 02 15:29:31.442786 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:31.442760    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a\": container with ID starting with ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a not found: ID does not exist" containerID="ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a"
Nov 02 15:29:31.442871 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.442796    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a"} err="failed to get container status \"ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a\": rpc error: code = NotFound desc = could not find container \"ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a\": container with ID starting with ad370f049cdb059daef9ebbc394328672be6772415130fcdf6c459be2116039a not found: ID does not exist"
Nov 02 15:29:31.535619 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.535528    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"]
Nov 02 15:29:31.540281 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.540255    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-22679"]
Nov 02 15:29:31.647306 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.647272    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"]
Nov 02 15:29:31.661909 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.661879    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-k62kg"]
Nov 02 15:29:31.772207 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.772171    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"]
Nov 02 15:29:31.778713 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.778688    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-rftjh"]
Nov 02 15:29:31.933734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.933654    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"]
Nov 02 15:29:31.954185 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:31.954160    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6dc69b-openshift-default-b67c99fc7-6hqmk"]
Nov 02 15:29:32.146549 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:32.146512    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"]
Nov 02 15:29:32.155949 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:32.155917    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-rm6ws"]
Nov 02 15:29:32.269943 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:32.269906    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"]
Nov 02 15:29:32.278899 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:32.278867    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-vdwrl"]
Nov 02 15:29:32.524574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:32.524494    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15bfe010-d72c-42f1-918c-8ec54254da96" path="/var/lib/kubelet/pods/15bfe010-d72c-42f1-918c-8ec54254da96/volumes"
Nov 02 15:29:32.525063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:32.525045    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40993be1-454f-4d83-8b5a-85c1975f71da" path="/var/lib/kubelet/pods/40993be1-454f-4d83-8b5a-85c1975f71da/volumes"
Nov 02 15:29:32.525603 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:32.525582    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" path="/var/lib/kubelet/pods/4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5/volumes"
Nov 02 15:29:32.526373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:32.526345    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" path="/var/lib/kubelet/pods/5ee0893d-8caa-4d67-971c-d51bd8955783/volumes"
Nov 02 15:29:32.526792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:32.526780    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" path="/var/lib/kubelet/pods/666c6a6f-4079-4d31-b6ff-1f2c1116a0ff/volumes"
Nov 02 15:29:32.527322 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:32.527293    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" path="/var/lib/kubelet/pods/806a03af-da93-4064-a70f-cc3ee68e5c9d/volumes"
Nov 02 15:29:33.197098 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.197054    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"]
Nov 02 15:29:33.246530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.246476    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-53fd96-49js8"]
Nov 02 15:29:33.647795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.647766    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:29:33.722067 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.722032    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istiod-ca-cert\") pod \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") "
Nov 02 15:29:33.722241 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.722083    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-podinfo\") pod \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") "
Nov 02 15:29:33.722241 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.722234    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-socket\") pod \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") "
Nov 02 15:29:33.722384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.722370    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-kube-api-access-qpgvm\" (UniqueName: \"kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-kube-api-access-qpgvm\") pod \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") "
Nov 02 15:29:33.722460 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.722425    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istiod-ca-cert" pod "6f4bd3cc-281d-491e-a0c5-8164b864ecc6" (UID: "6f4bd3cc-281d-491e-a0c5-8164b864ecc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:33.722460 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.722434    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-credential-socket\") pod \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") "
Nov 02 15:29:33.722571 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.722503    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-envoy\") pod \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") "
Nov 02 15:29:33.722571 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.722553    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-data\") pod \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") "
Nov 02 15:29:33.722669 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.722583    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-token\" (UniqueName: \"kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-token\") pod \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") "
Nov 02 15:29:33.722669 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.722647    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-certs\") pod \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\" (UID: \"6f4bd3cc-281d-491e-a0c5-8164b864ecc6\") "
Nov 02 15:29:33.723100 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.723029    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-data" pod "6f4bd3cc-281d-491e-a0c5-8164b864ecc6" (UID: "6f4bd3cc-281d-491e-a0c5-8164b864ecc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:33.723100 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.723043    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-socket" pod "6f4bd3cc-281d-491e-a0c5-8164b864ecc6" (UID: "6f4bd3cc-281d-491e-a0c5-8164b864ecc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:33.723354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.723302    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-credential-socket" pod "6f4bd3cc-281d-491e-a0c5-8164b864ecc6" (UID: "6f4bd3cc-281d-491e-a0c5-8164b864ecc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:33.723439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.723395    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:33.723439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.723416    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:33.723439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.723431    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:33.723613 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.723594    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-certs" pod "6f4bd3cc-281d-491e-a0c5-8164b864ecc6" (UID: "6f4bd3cc-281d-491e-a0c5-8164b864ecc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:33.726421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.725631    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-token" pod "6f4bd3cc-281d-491e-a0c5-8164b864ecc6" (UID: "6f4bd3cc-281d-491e-a0c5-8164b864ecc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:33.727574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.727550    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-podinfo" pod "6f4bd3cc-281d-491e-a0c5-8164b864ecc6" (UID: "6f4bd3cc-281d-491e-a0c5-8164b864ecc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:29:33.728112 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.728087    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-kube-api-access-qpgvm" pod "6f4bd3cc-281d-491e-a0c5-8164b864ecc6" (UID: "6f4bd3cc-281d-491e-a0c5-8164b864ecc6"). InnerVolumeSpecName "kube-api-access-qpgvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:33.730479 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.730450    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-envoy" pod "6f4bd3cc-281d-491e-a0c5-8164b864ecc6" (UID: "6f4bd3cc-281d-491e-a0c5-8164b864ecc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:33.826090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.826047    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpgvm\" (UniqueName: \"kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-kube-api-access-qpgvm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:33.826364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.826339    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:33.826521 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.826506    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:33.826619 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.826607    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:33.826713 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.826701    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:33.826835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:33.826819    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4bd3cc-281d-491e-a0c5-8164b864ecc6-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:34.398694 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:34.398659    4891 generic.go:356] "Generic (PLEG): container finished" podID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerID="f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50" exitCode=0
Nov 02 15:29:34.399085 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:34.398738    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"
Nov 02 15:29:34.399085 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:34.398743    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" event={"ID":"6f4bd3cc-281d-491e-a0c5-8164b864ecc6","Type":"ContainerDied","Data":"f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50"}
Nov 02 15:29:34.399085 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:34.398826    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5" event={"ID":"6f4bd3cc-281d-491e-a0c5-8164b864ecc6","Type":"ContainerDied","Data":"ad7ea1c1e30dd202de8f598010ade20c126546a13ead96038bf5dfaf6f798979"}
Nov 02 15:29:34.399085 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:34.398852    4891 scope.go:122] "RemoveContainer" containerID="f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50"
Nov 02 15:29:34.410609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:34.410592    4891 scope.go:122] "RemoveContainer" containerID="f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50"
Nov 02 15:29:34.410875 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:34.410855    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50\": container with ID starting with f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50 not found: ID does not exist" containerID="f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50"
Nov 02 15:29:34.410934 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:34.410886    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50"} err="failed to get container status \"f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50\": rpc error: code = NotFound desc = could not find container \"f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50\": container with ID starting with f6335122434838fb33144c03dfe890b9d3f00286616a640f5c5f5447ac7a5c50 not found: ID does not exist"
Nov 02 15:29:34.523887 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:34.523858    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f15ea9-ca87-40d6-941b-7d4f4e575a18" path="/var/lib/kubelet/pods/d3f15ea9-ca87-40d6-941b-7d4f4e575a18/volumes"
Nov 02 15:29:34.569441 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:34.569411    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"]
Nov 02 15:29:34.618935 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:34.618904    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-6tjz5"]
Nov 02 15:29:34.873773 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:34.873742    4891 scope.go:122] "RemoveContainer" containerID="695ad821b84b08580c481d1b4bd52ca2370a28a236d7ee6350f3473e1f26a070"
Nov 02 15:29:34.883977 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:34.883957    4891 scope.go:122] "RemoveContainer" containerID="ec623f024651514ad9e19c10156b3cac55d15f9f75dd1a964261f986b2af9062"
Nov 02 15:29:36.523695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:36.523660    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" path="/var/lib/kubelet/pods/6f4bd3cc-281d-491e-a0c5-8164b864ecc6/volumes"
Nov 02 15:29:41.390631 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.390593    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6773-a20785"]
Nov 02 15:29:41.391306 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391285    4891 cpu_manager.go:414] "containerMap: removing container" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy"
Nov 02 15:29:41.391306 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391322    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy"
Nov 02 15:29:41.391421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391334    4891 cpu_manager.go:414] "containerMap: removing container" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerName="istio-proxy"
Nov 02 15:29:41.391421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391340    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerName="istio-proxy"
Nov 02 15:29:41.391421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391348    4891 cpu_manager.go:414] "containerMap: removing container" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerName="istio-proxy"
Nov 02 15:29:41.391421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391353    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerName="istio-proxy"
Nov 02 15:29:41.391421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391368    4891 cpu_manager.go:414] "containerMap: removing container" podUID="d3f15ea9-ca87-40d6-941b-7d4f4e575a18" containerName="revoke-keys"
Nov 02 15:29:41.391421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391373    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f15ea9-ca87-40d6-941b-7d4f4e575a18" containerName="revoke-keys"
Nov 02 15:29:41.391421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391391    4891 cpu_manager.go:414] "containerMap: removing container" podUID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" containerName="istio-proxy"
Nov 02 15:29:41.391421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391395    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" containerName="istio-proxy"
Nov 02 15:29:41.391421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391408    4891 cpu_manager.go:414] "containerMap: removing container" podUID="15bfe010-d72c-42f1-918c-8ec54254da96" containerName="istio-proxy"
Nov 02 15:29:41.391421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391413    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15bfe010-d72c-42f1-918c-8ec54254da96" containerName="istio-proxy"
Nov 02 15:29:41.391421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391425    4891 cpu_manager.go:414] "containerMap: removing container" podUID="40993be1-454f-4d83-8b5a-85c1975f71da" containerName="istio-proxy"
Nov 02 15:29:41.391751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391430    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40993be1-454f-4d83-8b5a-85c1975f71da" containerName="istio-proxy"
Nov 02 15:29:41.391751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391441    4891 cpu_manager.go:414] "containerMap: removing container" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerName="istio-proxy"
Nov 02 15:29:41.391751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391446    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerName="istio-proxy"
Nov 02 15:29:41.391751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391540    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="666c6a6f-4079-4d31-b6ff-1f2c1116a0ff" containerName="istio-proxy"
Nov 02 15:29:41.391751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391548    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee0893d-8caa-4d67-971c-d51bd8955783" containerName="istio-proxy"
Nov 02 15:29:41.391751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391556    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4bd3cc-281d-491e-a0c5-8164b864ecc6" containerName="istio-proxy"
Nov 02 15:29:41.391751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391563    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f15ea9-ca87-40d6-941b-7d4f4e575a18" containerName="revoke-keys"
Nov 02 15:29:41.391751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391571    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d189f67-43b9-4d7c-88fe-25e1bdb6c2b5" containerName="istio-proxy"
Nov 02 15:29:41.391751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391579    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="15bfe010-d72c-42f1-918c-8ec54254da96" containerName="istio-proxy"
Nov 02 15:29:41.391751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391585    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="40993be1-454f-4d83-8b5a-85c1975f71da" containerName="istio-proxy"
Nov 02 15:29:41.391751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.391591    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="806a03af-da93-4064-a70f-cc3ee68e5c9d" containerName="istio-proxy"
Nov 02 15:29:41.396759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.396742    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6773-a20785"
Nov 02 15:29:41.450994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.450958    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6773-a20785"]
Nov 02 15:29:41.494761 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.494730    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xkc9h\""
Nov 02 15:29:41.527657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.527619    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcm2m\" (UniqueName: \"kubernetes.io/projected/c18a7bf1-78f0-408c-afcb-2fd21d1d8de4-kube-api-access-qcm2m\") pod \"test-curl-6773-a20785\" (UID: \"c18a7bf1-78f0-408c-afcb-2fd21d1d8de4\") " pod="opendatahub/test-curl-6773-a20785"
Nov 02 15:29:41.628802 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.628760    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcm2m\" (UniqueName: \"kubernetes.io/projected/c18a7bf1-78f0-408c-afcb-2fd21d1d8de4-kube-api-access-qcm2m\") pod \"test-curl-6773-a20785\" (UID: \"c18a7bf1-78f0-408c-afcb-2fd21d1d8de4\") " pod="opendatahub/test-curl-6773-a20785"
Nov 02 15:29:41.689432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.689360    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcm2m\" (UniqueName: \"kubernetes.io/projected/c18a7bf1-78f0-408c-afcb-2fd21d1d8de4-kube-api-access-qcm2m\") pod \"test-curl-6773-a20785\" (UID: \"c18a7bf1-78f0-408c-afcb-2fd21d1d8de4\") " pod="opendatahub/test-curl-6773-a20785"
Nov 02 15:29:41.709167 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:41.709139    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6773-a20785"
Nov 02 15:29:42.087792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:42.087750    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6773-a20785"]
Nov 02 15:29:42.089090 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:29:42.089063    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc18a7bf1_78f0_408c_afcb_2fd21d1d8de4.slice/crio-573e9e16c3b28ab7adc3f418f98eb0697ec66a44581a809d329ee1a73693a5c5 WatchSource:0}: Error finding container 573e9e16c3b28ab7adc3f418f98eb0697ec66a44581a809d329ee1a73693a5c5: Status 404 returned error can't find the container with id 573e9e16c3b28ab7adc3f418f98eb0697ec66a44581a809d329ee1a73693a5c5
Nov 02 15:29:42.091079 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:42.091060    4891 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:29:42.456764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:42.456674    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6773-a20785" event={"ID":"c18a7bf1-78f0-408c-afcb-2fd21d1d8de4","Type":"ContainerStarted","Data":"573e9e16c3b28ab7adc3f418f98eb0697ec66a44581a809d329ee1a73693a5c5"}
Nov 02 15:29:42.981001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:42.980965    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"]
Nov 02 15:29:42.987205 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:42.987185    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:43.039561 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.039524    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:29:43.089063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.089024    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:29:43.089063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.089036    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:29:43.089359 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.089184    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:29:43.093411 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.093385    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"]
Nov 02 15:29:43.149341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.149278    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd\" (UID: \"c376fb6b-a1be-4423-8510-163029504e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:43.149506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.149398    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd\" (UID: \"c376fb6b-a1be-4423-8510-163029504e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:43.149506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.149438    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzzwq\" (UniqueName: \"kubernetes.io/projected/c376fb6b-a1be-4423-8510-163029504e1d-kube-api-access-kzzwq\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd\" (UID: \"c376fb6b-a1be-4423-8510-163029504e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:43.250916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.250810    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd\" (UID: \"c376fb6b-a1be-4423-8510-163029504e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:43.250916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.250882    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd\" (UID: \"c376fb6b-a1be-4423-8510-163029504e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:43.250916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.250924    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzzwq\" (UniqueName: \"kubernetes.io/projected/c376fb6b-a1be-4423-8510-163029504e1d-kube-api-access-kzzwq\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd\" (UID: \"c376fb6b-a1be-4423-8510-163029504e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:43.251614 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.251587    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd\" (UID: \"c376fb6b-a1be-4423-8510-163029504e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:43.253255 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.253237    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd\" (UID: \"c376fb6b-a1be-4423-8510-163029504e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:43.323719 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.323688    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzzwq\" (UniqueName: \"kubernetes.io/projected/c376fb6b-a1be-4423-8510-163029504e1d-kube-api-access-kzzwq\") pod \"maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd\" (UID: \"c376fb6b-a1be-4423-8510-163029504e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:43.601555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.601525    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:43.998961 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:43.998930    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"]
Nov 02 15:29:43.999502 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:29:43.999463    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc376fb6b_a1be_4423_8510_163029504e1d.slice/crio-807e0f5a51858cc4fc1d36627329c03dc9b15dbd94745b4484b7790116be6f42 WatchSource:0}: Error finding container 807e0f5a51858cc4fc1d36627329c03dc9b15dbd94745b4484b7790116be6f42: Status 404 returned error can't find the container with id 807e0f5a51858cc4fc1d36627329c03dc9b15dbd94745b4484b7790116be6f42
Nov 02 15:29:44.281183 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:44.281141    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:29:44.478501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:44.478460    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd" event={"ID":"c376fb6b-a1be-4423-8510-163029504e1d","Type":"ContainerStarted","Data":"21708c0a154988a955268b1985fcc337ccbc76ae3cc9645173f520d67986c04b"}
Nov 02 15:29:44.478690 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:44.478527    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd" event={"ID":"c376fb6b-a1be-4423-8510-163029504e1d","Type":"ContainerStarted","Data":"807e0f5a51858cc4fc1d36627329c03dc9b15dbd94745b4484b7790116be6f42"}
Nov 02 15:29:44.605051 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:44.604985    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd" podStartSLOduration=2.604967452 podStartE2EDuration="2.604967452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:44.602563276 +0000 UTC m=+1992.195180075" watchObservedRunningTime="2026-11-02 15:29:44.604967452 +0000 UTC m=+1992.197584255"
Nov 02 15:29:45.490328 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.490284    4891 generic.go:356] "Generic (PLEG): container finished" podID="c376fb6b-a1be-4423-8510-163029504e1d" containerID="21708c0a154988a955268b1985fcc337ccbc76ae3cc9645173f520d67986c04b" exitCode=0
Nov 02 15:29:45.490494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.490370    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd" event={"ID":"c376fb6b-a1be-4423-8510-163029504e1d","Type":"ContainerDied","Data":"21708c0a154988a955268b1985fcc337ccbc76ae3cc9645173f520d67986c04b"}
Nov 02 15:29:45.492008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.491981    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6773-a20785" event={"ID":"c18a7bf1-78f0-408c-afcb-2fd21d1d8de4","Type":"ContainerStarted","Data":"1ba7ebf1608297d628f4248466dd9cfbf16666b5feb3ce8ea872918db99907fd"}
Nov 02 15:29:45.595822 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.595787    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"]
Nov 02 15:29:45.599862 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.599840    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:45.646360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.646275    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"]
Nov 02 15:29:45.688650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.688614    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr\" (UID: \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:45.688814 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.688787    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n97t\" (UniqueName: \"kubernetes.io/projected/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-kube-api-access-7n97t\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr\" (UID: \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:45.688862 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.688839    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr\" (UID: \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:45.790298 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.790263    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr\" (UID: \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:45.790540 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.790376    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n97t\" (UniqueName: \"kubernetes.io/projected/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-kube-api-access-7n97t\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr\" (UID: \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:45.790540 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.790409    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr\" (UID: \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:45.790916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.790893    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr\" (UID: \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:45.792767 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.792744    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr\" (UID: \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:45.838776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.838702    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6773-a20785" podStartSLOduration=1.9787554 podStartE2EDuration="4.838683595s" totalImagesPullingTime="2.859928195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:45.836621781 +0000 UTC m=+1993.429238582" watchObservedRunningTime="2026-11-02 15:29:45.838683595 +0000 UTC m=+1993.431300401"
Nov 02 15:29:45.851092 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.851062    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n97t\" (UniqueName: \"kubernetes.io/projected/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-kube-api-access-7n97t\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr\" (UID: \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:45.926211 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.926178    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:45.979562 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.979261    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"]
Nov 02 15:29:45.985719 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:45.985691    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:46.045168 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.045086    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"]
Nov 02 15:29:46.094625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.094592    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b\" (UID: \"1e34adf2-b639-4aee-863d-47cb29994a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:46.094808 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.094755    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b\" (UID: \"1e34adf2-b639-4aee-863d-47cb29994a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:46.094808 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.094801    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klvc7\" (UniqueName: \"kubernetes.io/projected/1e34adf2-b639-4aee-863d-47cb29994a2e-kube-api-access-klvc7\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b\" (UID: \"1e34adf2-b639-4aee-863d-47cb29994a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:46.196621 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.196588    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b\" (UID: \"1e34adf2-b639-4aee-863d-47cb29994a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:46.196790 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.196692    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b\" (UID: \"1e34adf2-b639-4aee-863d-47cb29994a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:46.196790 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.196727    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvc7\" (UniqueName: \"kubernetes.io/projected/1e34adf2-b639-4aee-863d-47cb29994a2e-kube-api-access-klvc7\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b\" (UID: \"1e34adf2-b639-4aee-863d-47cb29994a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:46.197393 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.197360    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b\" (UID: \"1e34adf2-b639-4aee-863d-47cb29994a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:46.199059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.199033    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b\" (UID: \"1e34adf2-b639-4aee-863d-47cb29994a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:46.258392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.258364    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvc7\" (UniqueName: \"kubernetes.io/projected/1e34adf2-b639-4aee-863d-47cb29994a2e-kube-api-access-klvc7\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b\" (UID: \"1e34adf2-b639-4aee-863d-47cb29994a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:46.304159 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.304082    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:46.307443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.307417    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"]
Nov 02 15:29:46.307941 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:29:46.307909    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c53929f_ef7b_43d2_841c_d6eda17b5aeb.slice/crio-738843fa2f5174ebf78189df45f21b75cadb2507236f7895d6d81195449386bc WatchSource:0}: Error finding container 738843fa2f5174ebf78189df45f21b75cadb2507236f7895d6d81195449386bc: Status 404 returned error can't find the container with id 738843fa2f5174ebf78189df45f21b75cadb2507236f7895d6d81195449386bc
Nov 02 15:29:46.500643 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.500604    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr" event={"ID":"3c53929f-ef7b-43d2-841c-d6eda17b5aeb","Type":"ContainerStarted","Data":"738843fa2f5174ebf78189df45f21b75cadb2507236f7895d6d81195449386bc"}
Nov 02 15:29:46.646217 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.646191    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:46.688044 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.688016    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"]
Nov 02 15:29:46.688769 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:29:46.688745    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e34adf2_b639_4aee_863d_47cb29994a2e.slice/crio-2b92808e21cbf57da5b5e8a19a9b6ffd97c7bf88281d83eb9d71dbe07fafc113 WatchSource:0}: Error finding container 2b92808e21cbf57da5b5e8a19a9b6ffd97c7bf88281d83eb9d71dbe07fafc113: Status 404 returned error can't find the container with id 2b92808e21cbf57da5b5e8a19a9b6ffd97c7bf88281d83eb9d71dbe07fafc113
Nov 02 15:29:46.804989 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.804894    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c376fb6b-a1be-4423-8510-163029504e1d-kube-api-access-kzzwq\" (UniqueName: \"kubernetes.io/projected/c376fb6b-a1be-4423-8510-163029504e1d-kube-api-access-kzzwq\") pod \"c376fb6b-a1be-4423-8510-163029504e1d\" (UID: \"c376fb6b-a1be-4423-8510-163029504e1d\") "
Nov 02 15:29:46.805143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.805082    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-openshift-ca\") pod \"c376fb6b-a1be-4423-8510-163029504e1d\" (UID: \"c376fb6b-a1be-4423-8510-163029504e1d\") "
Nov 02 15:29:46.805273 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.805183    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-xks-ca\") pod \"c376fb6b-a1be-4423-8510-163029504e1d\" (UID: \"c376fb6b-a1be-4423-8510-163029504e1d\") "
Nov 02 15:29:46.805541 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.805511    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-openshift-ca" pod "c376fb6b-a1be-4423-8510-163029504e1d" (UID: "c376fb6b-a1be-4423-8510-163029504e1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:46.805816 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.805798    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:46.807064 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.807038    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c376fb6b-a1be-4423-8510-163029504e1d-kube-api-access-kzzwq" pod "c376fb6b-a1be-4423-8510-163029504e1d" (UID: "c376fb6b-a1be-4423-8510-163029504e1d"). InnerVolumeSpecName "kube-api-access-kzzwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:46.807232 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.807213    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-xks-ca" pod "c376fb6b-a1be-4423-8510-163029504e1d" (UID: "c376fb6b-a1be-4423-8510-163029504e1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:29:46.907191 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.907160    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c376fb6b-a1be-4423-8510-163029504e1d-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:46.907191 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:46.907186    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzzwq\" (UniqueName: \"kubernetes.io/projected/c376fb6b-a1be-4423-8510-163029504e1d-kube-api-access-kzzwq\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:47.378413 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.378371    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6773-a20785"]
Nov 02 15:29:47.386601 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.386567    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6773-a20785"]
Nov 02 15:29:47.396036 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.394632    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"]
Nov 02 15:29:47.433492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.433453    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"]
Nov 02 15:29:47.510937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.510876    4891 generic.go:356] "Generic (PLEG): container finished" podID="3c53929f-ef7b-43d2-841c-d6eda17b5aeb" containerID="6e9504e9d4dc1571bab3e9193dd667bbce5dc80c298d5ad30a9e25f5a17f748f" exitCode=0
Nov 02 15:29:47.511116 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.510941    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr" event={"ID":"3c53929f-ef7b-43d2-841c-d6eda17b5aeb","Type":"ContainerDied","Data":"6e9504e9d4dc1571bab3e9193dd667bbce5dc80c298d5ad30a9e25f5a17f748f"}
Nov 02 15:29:47.513110 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.513073    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b" event={"ID":"1e34adf2-b639-4aee-863d-47cb29994a2e","Type":"ContainerStarted","Data":"3131e484adf020d393e56e9c40de2ae2986bc798525e5409486dd5c9bc4f0a69"}
Nov 02 15:29:47.513270 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.513121    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b" event={"ID":"1e34adf2-b639-4aee-863d-47cb29994a2e","Type":"ContainerStarted","Data":"2b92808e21cbf57da5b5e8a19a9b6ffd97c7bf88281d83eb9d71dbe07fafc113"}
Nov 02 15:29:47.516414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.516352    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"
Nov 02 15:29:47.516565 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.516346    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd" event={"ID":"c376fb6b-a1be-4423-8510-163029504e1d","Type":"ContainerDied","Data":"807e0f5a51858cc4fc1d36627329c03dc9b15dbd94745b4484b7790116be6f42"}
Nov 02 15:29:47.516565 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.516485    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="807e0f5a51858cc4fc1d36627329c03dc9b15dbd94745b4484b7790116be6f42"
Nov 02 15:29:47.518158 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.516790    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6773-a20785" podUID="c18a7bf1-78f0-408c-afcb-2fd21d1d8de4" containerName="test-curl-6773-a20785" containerID="cri-o://1ba7ebf1608297d628f4248466dd9cfbf16666b5feb3ce8ea872918db99907fd" gracePeriod=2
Nov 02 15:29:47.689232 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:47.689188    4891 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6773-a20785\" is forbidden: User \"system:node:ip-10-0-132-27.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-27.us-west-1.compute.internal' and this object" podUID="c18a7bf1-78f0-408c-afcb-2fd21d1d8de4" pod="opendatahub/test-curl-6773-a20785"
Nov 02 15:29:47.807420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:47.807359    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b" podStartSLOduration=2.807345656 podStartE2EDuration="2.807345656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:47.804509712 +0000 UTC m=+1995.397126512" watchObservedRunningTime="2026-11-02 15:29:47.807345656 +0000 UTC m=+1995.399962450"
Nov 02 15:29:48.022784 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:48.022750    4891 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6773-a20785\" is forbidden: User \"system:node:ip-10-0-132-27.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-27.us-west-1.compute.internal' and this object" podUID="c18a7bf1-78f0-408c-afcb-2fd21d1d8de4" pod="opendatahub/test-curl-6773-a20785"
Nov 02 15:29:48.279770 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.279644    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"]
Nov 02 15:29:48.281219 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.281186    4891 cpu_manager.go:414] "containerMap: removing container" podUID="c18a7bf1-78f0-408c-afcb-2fd21d1d8de4" containerName="test-curl-6773-a20785"
Nov 02 15:29:48.281367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.281218    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18a7bf1-78f0-408c-afcb-2fd21d1d8de4" containerName="test-curl-6773-a20785"
Nov 02 15:29:48.281367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.281326    4891 cpu_manager.go:414] "containerMap: removing container" podUID="c376fb6b-a1be-4423-8510-163029504e1d" containerName="revoke-keys"
Nov 02 15:29:48.281367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.281335    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c376fb6b-a1be-4423-8510-163029504e1d" containerName="revoke-keys"
Nov 02 15:29:48.282067 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.282044    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18a7bf1-78f0-408c-afcb-2fd21d1d8de4" containerName="test-curl-6773-a20785"
Nov 02 15:29:48.282156 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.282094    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="c376fb6b-a1be-4423-8510-163029504e1d" containerName="revoke-keys"
Nov 02 15:29:48.286989 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.286964    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:48.338667 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:48.338623    4891 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6773-a20785\" is forbidden: User \"system:node:ip-10-0-132-27.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-27.us-west-1.compute.internal' and this object" podUID="c18a7bf1-78f0-408c-afcb-2fd21d1d8de4" pod="opendatahub/test-curl-6773-a20785"
Nov 02 15:29:48.341863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.341830    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"]
Nov 02 15:29:48.427719 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.427687    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52\" (UID: \"66ebffac-d598-438e-830d-7fc9a70dc69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:48.427921 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.427896    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52\" (UID: \"66ebffac-d598-438e-830d-7fc9a70dc69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:48.427983 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.427971    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wvx\" (UniqueName: \"kubernetes.io/projected/66ebffac-d598-438e-830d-7fc9a70dc69a-kube-api-access-99wvx\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52\" (UID: \"66ebffac-d598-438e-830d-7fc9a70dc69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:48.528506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.528469    4891 generic.go:356] "Generic (PLEG): container finished" podID="1e34adf2-b639-4aee-863d-47cb29994a2e" containerID="3131e484adf020d393e56e9c40de2ae2986bc798525e5409486dd5c9bc4f0a69" exitCode=0
Nov 02 15:29:48.528691 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.528558    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b" event={"ID":"1e34adf2-b639-4aee-863d-47cb29994a2e","Type":"ContainerDied","Data":"3131e484adf020d393e56e9c40de2ae2986bc798525e5409486dd5c9bc4f0a69"}
Nov 02 15:29:48.528797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.528776    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52\" (UID: \"66ebffac-d598-438e-830d-7fc9a70dc69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:48.528860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.528843    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wvx\" (UniqueName: \"kubernetes.io/projected/66ebffac-d598-438e-830d-7fc9a70dc69a-kube-api-access-99wvx\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52\" (UID: \"66ebffac-d598-438e-830d-7fc9a70dc69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:48.528957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.528941    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52\" (UID: \"66ebffac-d598-438e-830d-7fc9a70dc69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:48.529808 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.529758    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52\" (UID: \"66ebffac-d598-438e-830d-7fc9a70dc69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:48.531259 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.531235    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52\" (UID: \"66ebffac-d598-438e-830d-7fc9a70dc69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:48.590099 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.590068    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wvx\" (UniqueName: \"kubernetes.io/projected/66ebffac-d598-438e-830d-7fc9a70dc69a-kube-api-access-99wvx\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52\" (UID: \"66ebffac-d598-438e-830d-7fc9a70dc69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:48.605451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.601896    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:48.716644 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.716610    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:48.837849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.837764    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-kube-api-access-7n97t\" (UniqueName: \"kubernetes.io/projected/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-kube-api-access-7n97t\") pod \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\" (UID: \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\") "
Nov 02 15:29:48.838026 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.837865    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-openshift-ca\") pod \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\" (UID: \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\") "
Nov 02 15:29:48.838119 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.838102    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-xks-ca\") pod \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\" (UID: \"3c53929f-ef7b-43d2-841c-d6eda17b5aeb\") "
Nov 02 15:29:48.838422 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.838376    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-openshift-ca" pod "3c53929f-ef7b-43d2-841c-d6eda17b5aeb" (UID: "3c53929f-ef7b-43d2-841c-d6eda17b5aeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:48.838816 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.838793    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:48.840641 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.840617    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-xks-ca" pod "3c53929f-ef7b-43d2-841c-d6eda17b5aeb" (UID: "3c53929f-ef7b-43d2-841c-d6eda17b5aeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:29:48.840814 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.840799    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-kube-api-access-7n97t" pod "3c53929f-ef7b-43d2-841c-d6eda17b5aeb" (UID: "3c53929f-ef7b-43d2-841c-d6eda17b5aeb"). InnerVolumeSpecName "kube-api-access-7n97t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:48.940442 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.940396    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:48.940442 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:48.940438    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n97t\" (UniqueName: \"kubernetes.io/projected/3c53929f-ef7b-43d2-841c-d6eda17b5aeb-kube-api-access-7n97t\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:49.046349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.046298    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"]
Nov 02 15:29:49.048503 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:29:49.048470    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ebffac_d598_438e_830d_7fc9a70dc69a.slice/crio-f94296bc49e56eedd5301273eedb4a89fbe87662407faa12ba76bb8732f210f2 WatchSource:0}: Error finding container f94296bc49e56eedd5301273eedb4a89fbe87662407faa12ba76bb8732f210f2: Status 404 returned error can't find the container with id f94296bc49e56eedd5301273eedb4a89fbe87662407faa12ba76bb8732f210f2
Nov 02 15:29:49.541936 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.541897    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52" event={"ID":"66ebffac-d598-438e-830d-7fc9a70dc69a","Type":"ContainerStarted","Data":"720ddceda902a0f55880fe157176a73f6f1c29f7d89ee958b990fdba6f0be913"}
Nov 02 15:29:49.542091 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.541969    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52" event={"ID":"66ebffac-d598-438e-830d-7fc9a70dc69a","Type":"ContainerStarted","Data":"f94296bc49e56eedd5301273eedb4a89fbe87662407faa12ba76bb8732f210f2"}
Nov 02 15:29:49.543833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.543805    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"
Nov 02 15:29:49.543978 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.543949    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr" event={"ID":"3c53929f-ef7b-43d2-841c-d6eda17b5aeb","Type":"ContainerDied","Data":"738843fa2f5174ebf78189df45f21b75cadb2507236f7895d6d81195449386bc"}
Nov 02 15:29:49.544041 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.543993    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="738843fa2f5174ebf78189df45f21b75cadb2507236f7895d6d81195449386bc"
Nov 02 15:29:49.661407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.661229    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52" podStartSLOduration=1.6612113320000002 podStartE2EDuration="1.661211332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:49.660952837 +0000 UTC m=+1997.253569633" watchObservedRunningTime="2026-11-02 15:29:49.661211332 +0000 UTC m=+1997.253828131"
Nov 02 15:29:49.771346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.771300    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:49.796123 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.796091    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6773-a20785"
Nov 02 15:29:49.954450 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.954358    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-openshift-ca\") pod \"1e34adf2-b639-4aee-863d-47cb29994a2e\" (UID: \"1e34adf2-b639-4aee-863d-47cb29994a2e\") "
Nov 02 15:29:49.954450 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.954430    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e34adf2-b639-4aee-863d-47cb29994a2e-kube-api-access-klvc7\" (UniqueName: \"kubernetes.io/projected/1e34adf2-b639-4aee-863d-47cb29994a2e-kube-api-access-klvc7\") pod \"1e34adf2-b639-4aee-863d-47cb29994a2e\" (UID: \"1e34adf2-b639-4aee-863d-47cb29994a2e\") "
Nov 02 15:29:49.954668 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.954470    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18a7bf1-78f0-408c-afcb-2fd21d1d8de4-kube-api-access-qcm2m\" (UniqueName: \"kubernetes.io/projected/c18a7bf1-78f0-408c-afcb-2fd21d1d8de4-kube-api-access-qcm2m\") pod \"c18a7bf1-78f0-408c-afcb-2fd21d1d8de4\" (UID: \"c18a7bf1-78f0-408c-afcb-2fd21d1d8de4\") "
Nov 02 15:29:49.954668 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.954574    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-xks-ca\") pod \"1e34adf2-b639-4aee-863d-47cb29994a2e\" (UID: \"1e34adf2-b639-4aee-863d-47cb29994a2e\") "
Nov 02 15:29:49.954815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.954782    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-openshift-ca" pod "1e34adf2-b639-4aee-863d-47cb29994a2e" (UID: "1e34adf2-b639-4aee-863d-47cb29994a2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:49.955327 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.955285    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:49.957114 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.957085    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-xks-ca" pod "1e34adf2-b639-4aee-863d-47cb29994a2e" (UID: "1e34adf2-b639-4aee-863d-47cb29994a2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:29:49.957589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.957563    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18a7bf1-78f0-408c-afcb-2fd21d1d8de4-kube-api-access-qcm2m" pod "c18a7bf1-78f0-408c-afcb-2fd21d1d8de4" (UID: "c18a7bf1-78f0-408c-afcb-2fd21d1d8de4"). InnerVolumeSpecName "kube-api-access-qcm2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:49.958844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:49.958824    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e34adf2-b639-4aee-863d-47cb29994a2e-kube-api-access-klvc7" pod "1e34adf2-b639-4aee-863d-47cb29994a2e" (UID: "1e34adf2-b639-4aee-863d-47cb29994a2e"). InnerVolumeSpecName "kube-api-access-klvc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:50.056865 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.056832    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klvc7\" (UniqueName: \"kubernetes.io/projected/1e34adf2-b639-4aee-863d-47cb29994a2e-kube-api-access-klvc7\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:50.056865 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.056869    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcm2m\" (UniqueName: \"kubernetes.io/projected/c18a7bf1-78f0-408c-afcb-2fd21d1d8de4-kube-api-access-qcm2m\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:50.057033 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.056886    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e34adf2-b639-4aee-863d-47cb29994a2e-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:50.526860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.526821    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18a7bf1-78f0-408c-afcb-2fd21d1d8de4" path="/var/lib/kubelet/pods/c18a7bf1-78f0-408c-afcb-2fd21d1d8de4/volumes"
Nov 02 15:29:50.552161 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.552129    4891 generic.go:356] "Generic (PLEG): container finished" podID="c18a7bf1-78f0-408c-afcb-2fd21d1d8de4" containerID="1ba7ebf1608297d628f4248466dd9cfbf16666b5feb3ce8ea872918db99907fd" exitCode=137
Nov 02 15:29:50.552324 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.552173    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6773-a20785"
Nov 02 15:29:50.552324 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.552229    4891 scope.go:122] "RemoveContainer" containerID="1ba7ebf1608297d628f4248466dd9cfbf16666b5feb3ce8ea872918db99907fd"
Nov 02 15:29:50.554124 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.554099    4891 generic.go:356] "Generic (PLEG): container finished" podID="66ebffac-d598-438e-830d-7fc9a70dc69a" containerID="720ddceda902a0f55880fe157176a73f6f1c29f7d89ee958b990fdba6f0be913" exitCode=0
Nov 02 15:29:50.554248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.554180    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52" event={"ID":"66ebffac-d598-438e-830d-7fc9a70dc69a","Type":"ContainerDied","Data":"720ddceda902a0f55880fe157176a73f6f1c29f7d89ee958b990fdba6f0be913"}
Nov 02 15:29:50.555845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.555828    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"
Nov 02 15:29:50.555962 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.555830    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b" event={"ID":"1e34adf2-b639-4aee-863d-47cb29994a2e","Type":"ContainerDied","Data":"2b92808e21cbf57da5b5e8a19a9b6ffd97c7bf88281d83eb9d71dbe07fafc113"}
Nov 02 15:29:50.555962 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.555930    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b92808e21cbf57da5b5e8a19a9b6ffd97c7bf88281d83eb9d71dbe07fafc113"
Nov 02 15:29:50.562753 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.562738    4891 scope.go:122] "RemoveContainer" containerID="1ba7ebf1608297d628f4248466dd9cfbf16666b5feb3ce8ea872918db99907fd"
Nov 02 15:29:50.562986 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:29:50.562969    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba7ebf1608297d628f4248466dd9cfbf16666b5feb3ce8ea872918db99907fd\": container with ID starting with 1ba7ebf1608297d628f4248466dd9cfbf16666b5feb3ce8ea872918db99907fd not found: ID does not exist" containerID="1ba7ebf1608297d628f4248466dd9cfbf16666b5feb3ce8ea872918db99907fd"
Nov 02 15:29:50.563039 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:50.562992    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba7ebf1608297d628f4248466dd9cfbf16666b5feb3ce8ea872918db99907fd"} err="failed to get container status \"1ba7ebf1608297d628f4248466dd9cfbf16666b5feb3ce8ea872918db99907fd\": rpc error: code = NotFound desc = could not find container \"1ba7ebf1608297d628f4248466dd9cfbf16666b5feb3ce8ea872918db99907fd\": container with ID starting with 1ba7ebf1608297d628f4248466dd9cfbf16666b5feb3ce8ea872918db99907fd not found: ID does not exist"
Nov 02 15:29:51.738733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:51.738698    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:51.779787 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:51.779752    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-openshift-ca\") pod \"66ebffac-d598-438e-830d-7fc9a70dc69a\" (UID: \"66ebffac-d598-438e-830d-7fc9a70dc69a\") "
Nov 02 15:29:51.779978 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:51.779816    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ebffac-d598-438e-830d-7fc9a70dc69a-kube-api-access-99wvx\" (UniqueName: \"kubernetes.io/projected/66ebffac-d598-438e-830d-7fc9a70dc69a-kube-api-access-99wvx\") pod \"66ebffac-d598-438e-830d-7fc9a70dc69a\" (UID: \"66ebffac-d598-438e-830d-7fc9a70dc69a\") "
Nov 02 15:29:51.779978 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:51.779874    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-xks-ca\") pod \"66ebffac-d598-438e-830d-7fc9a70dc69a\" (UID: \"66ebffac-d598-438e-830d-7fc9a70dc69a\") "
Nov 02 15:29:51.780189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:51.780164    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-openshift-ca" pod "66ebffac-d598-438e-830d-7fc9a70dc69a" (UID: "66ebffac-d598-438e-830d-7fc9a70dc69a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:51.781961 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:51.781927    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-xks-ca" pod "66ebffac-d598-438e-830d-7fc9a70dc69a" (UID: "66ebffac-d598-438e-830d-7fc9a70dc69a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:29:51.782065 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:51.782031    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ebffac-d598-438e-830d-7fc9a70dc69a-kube-api-access-99wvx" pod "66ebffac-d598-438e-830d-7fc9a70dc69a" (UID: "66ebffac-d598-438e-830d-7fc9a70dc69a"). InnerVolumeSpecName "kube-api-access-99wvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:51.881988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:51.881912    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:51.881988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:51.881944    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99wvx\" (UniqueName: \"kubernetes.io/projected/66ebffac-d598-438e-830d-7fc9a70dc69a-kube-api-access-99wvx\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:51.881988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:51.881954    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ebffac-d598-438e-830d-7fc9a70dc69a-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:29:52.572271 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:52.572241    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"
Nov 02 15:29:52.572477 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:52.572244    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52" event={"ID":"66ebffac-d598-438e-830d-7fc9a70dc69a","Type":"ContainerDied","Data":"f94296bc49e56eedd5301273eedb4a89fbe87662407faa12ba76bb8732f210f2"}
Nov 02 15:29:52.572477 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:52.572367    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f94296bc49e56eedd5301273eedb4a89fbe87662407faa12ba76bb8732f210f2"
Nov 02 15:29:55.743623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:55.743581    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"]
Nov 02 15:29:55.744087 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:55.743847    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy" containerID="cri-o://126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996" gracePeriod=30
Nov 02 15:29:58.643795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:58.643753    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"]
Nov 02 15:29:58.644423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:58.643986    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" podUID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" containerName="istio-proxy" containerID="cri-o://8196030bf19326e58de26e793a3a2f39c145d0de5b0fcf900a81a7e048e927b6" gracePeriod=30
Nov 02 15:29:59.555001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:59.554964    4891 patch_prober.go:29] interesting pod/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:29:59.555166 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:29:59.555030    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:30:00.190478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.190442    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"]
Nov 02 15:30:00.191050 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.191034    4891 cpu_manager.go:414] "containerMap: removing container" podUID="3c53929f-ef7b-43d2-841c-d6eda17b5aeb" containerName="revoke-keys"
Nov 02 15:30:00.191104 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.191051    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c53929f-ef7b-43d2-841c-d6eda17b5aeb" containerName="revoke-keys"
Nov 02 15:30:00.191104 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.191086    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1e34adf2-b639-4aee-863d-47cb29994a2e" containerName="revoke-keys"
Nov 02 15:30:00.191104 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.191091    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e34adf2-b639-4aee-863d-47cb29994a2e" containerName="revoke-keys"
Nov 02 15:30:00.191202 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.191104    4891 cpu_manager.go:414] "containerMap: removing container" podUID="66ebffac-d598-438e-830d-7fc9a70dc69a" containerName="revoke-keys"
Nov 02 15:30:00.191202 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.191109    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ebffac-d598-438e-830d-7fc9a70dc69a" containerName="revoke-keys"
Nov 02 15:30:00.191263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.191215    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e34adf2-b639-4aee-863d-47cb29994a2e" containerName="revoke-keys"
Nov 02 15:30:00.191263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.191227    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ebffac-d598-438e-830d-7fc9a70dc69a" containerName="revoke-keys"
Nov 02 15:30:00.191263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.191236    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c53929f-ef7b-43d2-841c-d6eda17b5aeb" containerName="revoke-keys"
Nov 02 15:30:00.197279 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.197255    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"]
Nov 02 15:30:00.197436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.197409    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"
Nov 02 15:30:00.201788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.201765    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"]
Nov 02 15:30:00.201908 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.201889    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"
Nov 02 15:30:00.207594 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.207568    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"
Nov 02 15:30:00.235533 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.235510    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"]
Nov 02 15:30:00.239838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.239817    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"]
Nov 02 15:30:00.239962 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.239932    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"
Nov 02 15:30:00.244019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.244000    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"]
Nov 02 15:30:00.244146 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.244120    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"
Nov 02 15:30:00.248282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.248264    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"]
Nov 02 15:30:00.248416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.248401    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"
Nov 02 15:30:00.253094 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.253078    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"
Nov 02 15:30:00.263700 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.263673    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"]
Nov 02 15:30:00.269528 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.269500    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"]
Nov 02 15:30:00.280938 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.280893    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:30:00.281880 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.281781    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:30:00.282039 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.281887    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:30:00.288094 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.288064    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"]
Nov 02 15:30:00.293382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.293356    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"]
Nov 02 15:30:00.297728 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.297704    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"]
Nov 02 15:30:00.298768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.298742    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"
Nov 02 15:30:00.302718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.302698    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"
Nov 02 15:30:00.314728 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.314627    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"]
Nov 02 15:30:00.321924 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.321892    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"]
Nov 02 15:30:00.322040 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.322020    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"
Nov 02 15:30:00.324872 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.324852    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"]
Nov 02 15:30:00.328370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328345    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d692\" (UniqueName: \"kubernetes.io/projected/f33e9cd6-6338-4897-9adc-8282a614e231-kube-api-access-4d692\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6\" (UID: \"f33e9cd6-6338-4897-9adc-8282a614e231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"
Nov 02 15:30:00.328494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328405    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c01d26-e151-44a6-8501-d9618ad61ab9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9\" (UID: \"96c01d26-e151-44a6-8501-d9618ad61ab9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"
Nov 02 15:30:00.328594 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328569    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33e9cd6-6338-4897-9adc-8282a614e231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6\" (UID: \"f33e9cd6-6338-4897-9adc-8282a614e231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"
Nov 02 15:30:00.328674 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328633    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-659jr\" (UniqueName: \"kubernetes.io/projected/585afb52-1760-4ed3-8fbb-e0e5e3077372-kube-api-access-659jr\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7\" (UID: \"585afb52-1760-4ed3-8fbb-e0e5e3077372\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"
Nov 02 15:30:00.328747 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328672    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/585afb52-1760-4ed3-8fbb-e0e5e3077372-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7\" (UID: \"585afb52-1760-4ed3-8fbb-e0e5e3077372\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"
Nov 02 15:30:00.328747 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328703    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddzz2\" (UniqueName: \"kubernetes.io/projected/84777af8-4c53-4f7e-9a1a-940e6798353e-kube-api-access-ddzz2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb\" (UID: \"84777af8-4c53-4f7e-9a1a-940e6798353e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"
Nov 02 15:30:00.328910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328754    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mshbf\" (UniqueName: \"kubernetes.io/projected/96c01d26-e151-44a6-8501-d9618ad61ab9-kube-api-access-mshbf\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9\" (UID: \"96c01d26-e151-44a6-8501-d9618ad61ab9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"
Nov 02 15:30:00.328910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328787    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwtxg\" (UniqueName: \"kubernetes.io/projected/de5a0c88-2c43-48b1-b966-edb2603ef847-kube-api-access-zwtxg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls\" (UID: \"de5a0c88-2c43-48b1-b966-edb2603ef847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"
Nov 02 15:30:00.328910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328835    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84777af8-4c53-4f7e-9a1a-940e6798353e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb\" (UID: \"84777af8-4c53-4f7e-9a1a-940e6798353e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"
Nov 02 15:30:00.328910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328863    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49j95\" (UniqueName: \"kubernetes.io/projected/d59415e5-4268-42af-b993-3eea7e047ef5-kube-api-access-49j95\") pod \"maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp\" (UID: \"d59415e5-4268-42af-b993-3eea7e047ef5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"
Nov 02 15:30:00.328910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328889    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e212aa5a-ae15-4ac4-8958-d38c83b2658f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb\" (UID: \"e212aa5a-ae15-4ac4-8958-d38c83b2658f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"
Nov 02 15:30:00.329424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328940    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5a0c88-2c43-48b1-b966-edb2603ef847-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls\" (UID: \"de5a0c88-2c43-48b1-b966-edb2603ef847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"
Nov 02 15:30:00.329424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.328964    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59415e5-4268-42af-b993-3eea7e047ef5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp\" (UID: \"d59415e5-4268-42af-b993-3eea7e047ef5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"
Nov 02 15:30:00.329424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.329043    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26pvv\" (UniqueName: \"kubernetes.io/projected/e212aa5a-ae15-4ac4-8958-d38c83b2658f-kube-api-access-26pvv\") pod \"maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb\" (UID: \"e212aa5a-ae15-4ac4-8958-d38c83b2658f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"
Nov 02 15:30:00.330820 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.330803    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"]
Nov 02 15:30:00.330943 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.330922    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"
Nov 02 15:30:00.343337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.339923    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"
Nov 02 15:30:00.350290 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.350264    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"]
Nov 02 15:30:00.360370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.360283    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"]
Nov 02 15:30:00.369232 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.369203    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"]
Nov 02 15:30:00.379171 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.379141    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"]
Nov 02 15:30:00.408443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.408415    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"]
Nov 02 15:30:00.412574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.412547    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"]
Nov 02 15:30:00.415191 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.415167    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"]
Nov 02 15:30:00.426831 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.426802    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"]
Nov 02 15:30:00.431668 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.431639    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9hsq\" (UniqueName: \"kubernetes.io/projected/db707186-6796-4dad-893d-cb8ced999821-kube-api-access-t9hsq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9\" (UID: \"db707186-6796-4dad-893d-cb8ced999821\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"
Nov 02 15:30:00.431869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.431852    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qcs8\" (UniqueName: \"kubernetes.io/projected/9a174897-47d1-4a4c-994a-7186e7e405ef-kube-api-access-8qcs8\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg\" (UID: \"9a174897-47d1-4a4c-994a-7186e7e405ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"
Nov 02 15:30:00.432009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.431992    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-659jr\" (UniqueName: \"kubernetes.io/projected/585afb52-1760-4ed3-8fbb-e0e5e3077372-kube-api-access-659jr\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7\" (UID: \"585afb52-1760-4ed3-8fbb-e0e5e3077372\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"
Nov 02 15:30:00.432419 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432397    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/585afb52-1760-4ed3-8fbb-e0e5e3077372-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7\" (UID: \"585afb52-1760-4ed3-8fbb-e0e5e3077372\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"
Nov 02 15:30:00.432530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432434    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fba93da-7b0c-4456-8c69-461cfbb3fca9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd\" (UID: \"9fba93da-7b0c-4456-8c69-461cfbb3fca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"
Nov 02 15:30:00.432530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432465    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddzz2\" (UniqueName: \"kubernetes.io/projected/84777af8-4c53-4f7e-9a1a-940e6798353e-kube-api-access-ddzz2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb\" (UID: \"84777af8-4c53-4f7e-9a1a-940e6798353e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"
Nov 02 15:30:00.432530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432523    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mshbf\" (UniqueName: \"kubernetes.io/projected/96c01d26-e151-44a6-8501-d9618ad61ab9-kube-api-access-mshbf\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9\" (UID: \"96c01d26-e151-44a6-8501-d9618ad61ab9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"
Nov 02 15:30:00.432727 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432563    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwtxg\" (UniqueName: \"kubernetes.io/projected/de5a0c88-2c43-48b1-b966-edb2603ef847-kube-api-access-zwtxg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls\" (UID: \"de5a0c88-2c43-48b1-b966-edb2603ef847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"
Nov 02 15:30:00.432727 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432625    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84777af8-4c53-4f7e-9a1a-940e6798353e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb\" (UID: \"84777af8-4c53-4f7e-9a1a-940e6798353e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"
Nov 02 15:30:00.432727 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432650    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49j95\" (UniqueName: \"kubernetes.io/projected/d59415e5-4268-42af-b993-3eea7e047ef5-kube-api-access-49j95\") pod \"maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp\" (UID: \"d59415e5-4268-42af-b993-3eea7e047ef5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"
Nov 02 15:30:00.432727 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432680    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e212aa5a-ae15-4ac4-8958-d38c83b2658f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb\" (UID: \"e212aa5a-ae15-4ac4-8958-d38c83b2658f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"
Nov 02 15:30:00.432727 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432711    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5wwm\" (UniqueName: \"kubernetes.io/projected/2ae86072-ec09-40f3-93f1-452f118684ba-kube-api-access-m5wwm\") pod \"maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb\" (UID: \"2ae86072-ec09-40f3-93f1-452f118684ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"
Nov 02 15:30:00.433045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432769    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5a0c88-2c43-48b1-b966-edb2603ef847-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls\" (UID: \"de5a0c88-2c43-48b1-b966-edb2603ef847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"
Nov 02 15:30:00.433045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432856    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59415e5-4268-42af-b993-3eea7e047ef5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp\" (UID: \"d59415e5-4268-42af-b993-3eea7e047ef5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"
Nov 02 15:30:00.433045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432906    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db707186-6796-4dad-893d-cb8ced999821-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9\" (UID: \"db707186-6796-4dad-893d-cb8ced999821\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"
Nov 02 15:30:00.433045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.432951    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a174897-47d1-4a4c-994a-7186e7e405ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg\" (UID: \"9a174897-47d1-4a4c-994a-7186e7e405ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"
Nov 02 15:30:00.433045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.433007    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26pvv\" (UniqueName: \"kubernetes.io/projected/e212aa5a-ae15-4ac4-8958-d38c83b2658f-kube-api-access-26pvv\") pod \"maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb\" (UID: \"e212aa5a-ae15-4ac4-8958-d38c83b2658f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"
Nov 02 15:30:00.433437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.433064    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdw2\" (UniqueName: \"kubernetes.io/projected/9fba93da-7b0c-4456-8c69-461cfbb3fca9-kube-api-access-gjdw2\") pod \"maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd\" (UID: \"9fba93da-7b0c-4456-8c69-461cfbb3fca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"
Nov 02 15:30:00.433437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.433115    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhdrf\" (UniqueName: \"kubernetes.io/projected/13461dec-71c7-4ffc-aabb-f84f820c4502-kube-api-access-bhdrf\") pod \"maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6\" (UID: \"13461dec-71c7-4ffc-aabb-f84f820c4502\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"
Nov 02 15:30:00.433437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.433155    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae86072-ec09-40f3-93f1-452f118684ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb\" (UID: \"2ae86072-ec09-40f3-93f1-452f118684ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"
Nov 02 15:30:00.433437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.433190    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d692\" (UniqueName: \"kubernetes.io/projected/f33e9cd6-6338-4897-9adc-8282a614e231-kube-api-access-4d692\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6\" (UID: \"f33e9cd6-6338-4897-9adc-8282a614e231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"
Nov 02 15:30:00.433437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.433257    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c01d26-e151-44a6-8501-d9618ad61ab9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9\" (UID: \"96c01d26-e151-44a6-8501-d9618ad61ab9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"
Nov 02 15:30:00.433437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.433323    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33e9cd6-6338-4897-9adc-8282a614e231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6\" (UID: \"f33e9cd6-6338-4897-9adc-8282a614e231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"
Nov 02 15:30:00.433437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.433350    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13461dec-71c7-4ffc-aabb-f84f820c4502-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6\" (UID: \"13461dec-71c7-4ffc-aabb-f84f820c4502\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"
Nov 02 15:30:00.433913 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.433885    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/585afb52-1760-4ed3-8fbb-e0e5e3077372-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7\" (UID: \"585afb52-1760-4ed3-8fbb-e0e5e3077372\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"
Nov 02 15:30:00.435793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.434165    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84777af8-4c53-4f7e-9a1a-940e6798353e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb\" (UID: \"84777af8-4c53-4f7e-9a1a-940e6798353e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"
Nov 02 15:30:00.436482 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.434841    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59415e5-4268-42af-b993-3eea7e047ef5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp\" (UID: \"d59415e5-4268-42af-b993-3eea7e047ef5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"
Nov 02 15:30:00.436716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.435486    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e212aa5a-ae15-4ac4-8958-d38c83b2658f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb\" (UID: \"e212aa5a-ae15-4ac4-8958-d38c83b2658f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"
Nov 02 15:30:00.436846 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.436246    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33e9cd6-6338-4897-9adc-8282a614e231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6\" (UID: \"f33e9cd6-6338-4897-9adc-8282a614e231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"
Nov 02 15:30:00.436964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.436433    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5a0c88-2c43-48b1-b966-edb2603ef847-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls\" (UID: \"de5a0c88-2c43-48b1-b966-edb2603ef847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"
Nov 02 15:30:00.436964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.436861    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c01d26-e151-44a6-8501-d9618ad61ab9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9\" (UID: \"96c01d26-e151-44a6-8501-d9618ad61ab9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"
Nov 02 15:30:00.497975 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.497873    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-659jr\" (UniqueName: \"kubernetes.io/projected/585afb52-1760-4ed3-8fbb-e0e5e3077372-kube-api-access-659jr\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7\" (UID: \"585afb52-1760-4ed3-8fbb-e0e5e3077372\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"
Nov 02 15:30:00.498215 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.497996    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwtxg\" (UniqueName: \"kubernetes.io/projected/de5a0c88-2c43-48b1-b966-edb2603ef847-kube-api-access-zwtxg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls\" (UID: \"de5a0c88-2c43-48b1-b966-edb2603ef847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"
Nov 02 15:30:00.498379 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.498272    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddzz2\" (UniqueName: \"kubernetes.io/projected/84777af8-4c53-4f7e-9a1a-940e6798353e-kube-api-access-ddzz2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb\" (UID: \"84777af8-4c53-4f7e-9a1a-940e6798353e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"
Nov 02 15:30:00.498718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.498698    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49j95\" (UniqueName: \"kubernetes.io/projected/d59415e5-4268-42af-b993-3eea7e047ef5-kube-api-access-49j95\") pod \"maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp\" (UID: \"d59415e5-4268-42af-b993-3eea7e047ef5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"
Nov 02 15:30:00.499849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.499821    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mshbf\" (UniqueName: \"kubernetes.io/projected/96c01d26-e151-44a6-8501-d9618ad61ab9-kube-api-access-mshbf\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9\" (UID: \"96c01d26-e151-44a6-8501-d9618ad61ab9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"
Nov 02 15:30:00.500250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.500228    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26pvv\" (UniqueName: \"kubernetes.io/projected/e212aa5a-ae15-4ac4-8958-d38c83b2658f-kube-api-access-26pvv\") pod \"maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb\" (UID: \"e212aa5a-ae15-4ac4-8958-d38c83b2658f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"
Nov 02 15:30:00.503104 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.503078    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d692\" (UniqueName: \"kubernetes.io/projected/f33e9cd6-6338-4897-9adc-8282a614e231-kube-api-access-4d692\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6\" (UID: \"f33e9cd6-6338-4897-9adc-8282a614e231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"
Nov 02 15:30:00.519450 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.515285    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"
Nov 02 15:30:00.529988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.529944    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"
Nov 02 15:30:00.533186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.531230    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"
Nov 02 15:30:00.534353 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.534283    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhdrf\" (UniqueName: \"kubernetes.io/projected/13461dec-71c7-4ffc-aabb-f84f820c4502-kube-api-access-bhdrf\") pod \"maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6\" (UID: \"13461dec-71c7-4ffc-aabb-f84f820c4502\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"
Nov 02 15:30:00.534511 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.534372    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae86072-ec09-40f3-93f1-452f118684ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb\" (UID: \"2ae86072-ec09-40f3-93f1-452f118684ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"
Nov 02 15:30:00.534806 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.534436    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13461dec-71c7-4ffc-aabb-f84f820c4502-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6\" (UID: \"13461dec-71c7-4ffc-aabb-f84f820c4502\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"
Nov 02 15:30:00.534806 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.534714    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hsq\" (UniqueName: \"kubernetes.io/projected/db707186-6796-4dad-893d-cb8ced999821-kube-api-access-t9hsq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9\" (UID: \"db707186-6796-4dad-893d-cb8ced999821\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"
Nov 02 15:30:00.534806 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.534760    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qcs8\" (UniqueName: \"kubernetes.io/projected/9a174897-47d1-4a4c-994a-7186e7e405ef-kube-api-access-8qcs8\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg\" (UID: \"9a174897-47d1-4a4c-994a-7186e7e405ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"
Nov 02 15:30:00.534806 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.534804    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fba93da-7b0c-4456-8c69-461cfbb3fca9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd\" (UID: \"9fba93da-7b0c-4456-8c69-461cfbb3fca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"
Nov 02 15:30:00.535096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.534899    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5wwm\" (UniqueName: \"kubernetes.io/projected/2ae86072-ec09-40f3-93f1-452f118684ba-kube-api-access-m5wwm\") pod \"maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb\" (UID: \"2ae86072-ec09-40f3-93f1-452f118684ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"
Nov 02 15:30:00.535096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.534959    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db707186-6796-4dad-893d-cb8ced999821-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9\" (UID: \"db707186-6796-4dad-893d-cb8ced999821\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"
Nov 02 15:30:00.535096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.534995    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a174897-47d1-4a4c-994a-7186e7e405ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg\" (UID: \"9a174897-47d1-4a4c-994a-7186e7e405ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"
Nov 02 15:30:00.535096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.535062    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdw2\" (UniqueName: \"kubernetes.io/projected/9fba93da-7b0c-4456-8c69-461cfbb3fca9-kube-api-access-gjdw2\") pod \"maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd\" (UID: \"9fba93da-7b0c-4456-8c69-461cfbb3fca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"
Nov 02 15:30:00.539461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.536550    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae86072-ec09-40f3-93f1-452f118684ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb\" (UID: \"2ae86072-ec09-40f3-93f1-452f118684ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"
Nov 02 15:30:00.539461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.537125    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fba93da-7b0c-4456-8c69-461cfbb3fca9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd\" (UID: \"9fba93da-7b0c-4456-8c69-461cfbb3fca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"
Nov 02 15:30:00.539461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.537432    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db707186-6796-4dad-893d-cb8ced999821-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9\" (UID: \"db707186-6796-4dad-893d-cb8ced999821\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"
Nov 02 15:30:00.539461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.537731    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13461dec-71c7-4ffc-aabb-f84f820c4502-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6\" (UID: \"13461dec-71c7-4ffc-aabb-f84f820c4502\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"
Nov 02 15:30:00.539461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.538030    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a174897-47d1-4a4c-994a-7186e7e405ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg\" (UID: \"9a174897-47d1-4a4c-994a-7186e7e405ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"
Nov 02 15:30:00.555991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.555965    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"
Nov 02 15:30:00.565772 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.565745    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"
Nov 02 15:30:00.582037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.581349    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"
Nov 02 15:30:00.592252 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.592197    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"
Nov 02 15:30:00.602015 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.601986    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhdrf\" (UniqueName: \"kubernetes.io/projected/13461dec-71c7-4ffc-aabb-f84f820c4502-kube-api-access-bhdrf\") pod \"maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6\" (UID: \"13461dec-71c7-4ffc-aabb-f84f820c4502\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"
Nov 02 15:30:00.603198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.603170    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qcs8\" (UniqueName: \"kubernetes.io/projected/9a174897-47d1-4a4c-994a-7186e7e405ef-kube-api-access-8qcs8\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg\" (UID: \"9a174897-47d1-4a4c-994a-7186e7e405ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"
Nov 02 15:30:00.604874 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.604621    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5wwm\" (UniqueName: \"kubernetes.io/projected/2ae86072-ec09-40f3-93f1-452f118684ba-kube-api-access-m5wwm\") pod \"maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb\" (UID: \"2ae86072-ec09-40f3-93f1-452f118684ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"
Nov 02 15:30:00.605630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.605593    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hsq\" (UniqueName: \"kubernetes.io/projected/db707186-6796-4dad-893d-cb8ced999821-kube-api-access-t9hsq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9\" (UID: \"db707186-6796-4dad-893d-cb8ced999821\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"
Nov 02 15:30:00.612727 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.612678    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdw2\" (UniqueName: \"kubernetes.io/projected/9fba93da-7b0c-4456-8c69-461cfbb3fca9-kube-api-access-gjdw2\") pod \"maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd\" (UID: \"9fba93da-7b0c-4456-8c69-461cfbb3fca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"
Nov 02 15:30:00.630493 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.630219    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"
Nov 02 15:30:00.640803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.640572    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"
Nov 02 15:30:00.698504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.698136    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"
Nov 02 15:30:00.713886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.713666    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"
Nov 02 15:30:00.724324 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.719008    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"
Nov 02 15:30:00.962440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:00.958851    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"]
Nov 02 15:30:00.962440 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:30:00.961881    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84777af8_4c53_4f7e_9a1a_940e6798353e.slice/crio-5bd2418f4b78fa0dd516ef33324174346f77d5ac13801b8db1b6ca06720e0de8 WatchSource:0}: Error finding container 5bd2418f4b78fa0dd516ef33324174346f77d5ac13801b8db1b6ca06720e0de8: Status 404 returned error can't find the container with id 5bd2418f4b78fa0dd516ef33324174346f77d5ac13801b8db1b6ca06720e0de8
Nov 02 15:30:01.159502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.159477    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:30:01.184813 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.184780    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-socket\") pod \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") "
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.184895    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-kube-api-access-bcd7c\" (UniqueName: \"kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-kube-api-access-bcd7c\") pod \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") "
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.184943    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-data\") pod \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") "
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.185018    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-envoy\") pod \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") "
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.185075    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-socket" pod "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" (UID: "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.185131    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-certs\") pod \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") "
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.185213    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-credential-socket\") pod \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") "
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.185260    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-token\" (UniqueName: \"kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-token\") pod \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") "
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.185295    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-data" pod "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" (UID: "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.185307    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-podinfo\") pod \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") "
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.185393    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istiod-ca-cert\") pod \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\" (UID: \"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5\") "
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.185500    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-credential-socket" pod "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" (UID: "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.185791    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-certs" pod "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" (UID: "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.186111    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istiod-ca-cert" pod "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" (UID: "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.186194    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.186217    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.186232    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.186248    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:01.190622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.188922    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-envoy" pod "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" (UID: "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:01.209982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.199543    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-kube-api-access-bcd7c" pod "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" (UID: "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5"). InnerVolumeSpecName "kube-api-access-bcd7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:01.211659 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.211598    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-podinfo" pod "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" (UID: "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:30:01.308139 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.304648    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:01.308139 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.304675    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:01.308139 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.304690    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcd7c\" (UniqueName: \"kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-kube-api-access-bcd7c\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:01.308139 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.304706    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:01.320469 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.318822    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-token" pod "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" (UID: "47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:01.324955 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.324871    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"]
Nov 02 15:30:01.326300 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:30:01.326265    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585afb52_1760_4ed3_8fbb_e0e5e3077372.slice/crio-536e05acb6fd9736f694805077809fc1174ec76f759ce6b5d39b12e03f0c2371 WatchSource:0}: Error finding container 536e05acb6fd9736f694805077809fc1174ec76f759ce6b5d39b12e03f0c2371: Status 404 returned error can't find the container with id 536e05acb6fd9736f694805077809fc1174ec76f759ce6b5d39b12e03f0c2371
Nov 02 15:30:01.332224 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.332193    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"]
Nov 02 15:30:01.405868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.405837    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:01.406626 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.406599    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"]
Nov 02 15:30:01.407835 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:30:01.407802    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33e9cd6_6338_4897_9adc_8282a614e231.slice/crio-af8ea3c1c889f3abfe967f5eb86dff327c4092414acaa60f37be66a1e146a417 WatchSource:0}: Error finding container af8ea3c1c889f3abfe967f5eb86dff327c4092414acaa60f37be66a1e146a417: Status 404 returned error can't find the container with id af8ea3c1c889f3abfe967f5eb86dff327c4092414acaa60f37be66a1e146a417
Nov 02 15:30:01.509092 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.509057    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"]
Nov 02 15:30:01.517665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.516728    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"]
Nov 02 15:30:01.531604 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.531573    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"]
Nov 02 15:30:01.538543 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.538136    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"]
Nov 02 15:30:01.575539 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.573952    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"]
Nov 02 15:30:01.611046 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.610939    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"]
Nov 02 15:30:01.614894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.614871    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"]
Nov 02 15:30:01.648188 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:30:01.647364    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode212aa5a_ae15_4ac4_8958_d38c83b2658f.slice/crio-5935660c591ecfb7fcc2b3987ff8577febcd00558314bb2feaad14ccba3e0999 WatchSource:0}: Error finding container 5935660c591ecfb7fcc2b3987ff8577febcd00558314bb2feaad14ccba3e0999: Status 404 returned error can't find the container with id 5935660c591ecfb7fcc2b3987ff8577febcd00558314bb2feaad14ccba3e0999
Nov 02 15:30:01.691756 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.691720    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"]
Nov 02 15:30:01.754789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.738553    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp" event={"ID":"d59415e5-4268-42af-b993-3eea7e047ef5","Type":"ContainerStarted","Data":"239f4db1e5f16df4dcdab3a90126e097bad9fd3db4dc06c6465dce73229adff8"}
Nov 02 15:30:01.754789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.740964    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9" event={"ID":"96c01d26-e151-44a6-8501-d9618ad61ab9","Type":"ContainerStarted","Data":"7e61229be1b17ebcc8e141a52aab20f20e1a86726fb89f7b0b3d77184a597cf4"}
Nov 02 15:30:01.754789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.751765    4891 generic.go:356] "Generic (PLEG): container finished" podID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerID="126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996" exitCode=0
Nov 02 15:30:01.754789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.751842    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" event={"ID":"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5","Type":"ContainerDied","Data":"126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996"}
Nov 02 15:30:01.754789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.751881    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl" event={"ID":"47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5","Type":"ContainerDied","Data":"fc3005481bcbc1f8ee21c8ca4d10394ed60261fac10998dc61fe940db3a37655"}
Nov 02 15:30:01.754789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.751904    4891 scope.go:122] "RemoveContainer" containerID="126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996"
Nov 02 15:30:01.754789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.752034    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"
Nov 02 15:30:01.764921 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:30:01.764891    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb707186_6796_4dad_893d_cb8ced999821.slice/crio-b9d50b5f983af2caca29e4c1229b67acbd194e29cdc3816d7df4b383e28c055b WatchSource:0}: Error finding container b9d50b5f983af2caca29e4c1229b67acbd194e29cdc3816d7df4b383e28c055b: Status 404 returned error can't find the container with id b9d50b5f983af2caca29e4c1229b67acbd194e29cdc3816d7df4b383e28c055b
Nov 02 15:30:01.769374 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.769343    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb" event={"ID":"84777af8-4c53-4f7e-9a1a-940e6798353e","Type":"ContainerStarted","Data":"522937bcdb07377a02b1fee17e0a3b57bab0ce13bc75ffaf800adfdb03d4c872"}
Nov 02 15:30:01.769495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.769404    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb" event={"ID":"84777af8-4c53-4f7e-9a1a-940e6798353e","Type":"ContainerStarted","Data":"5bd2418f4b78fa0dd516ef33324174346f77d5ac13801b8db1b6ca06720e0de8"}
Nov 02 15:30:01.771810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.771780    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb" event={"ID":"e212aa5a-ae15-4ac4-8958-d38c83b2658f","Type":"ContainerStarted","Data":"5935660c591ecfb7fcc2b3987ff8577febcd00558314bb2feaad14ccba3e0999"}
Nov 02 15:30:01.773237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.773210    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls" event={"ID":"de5a0c88-2c43-48b1-b966-edb2603ef847","Type":"ContainerStarted","Data":"3a2134eb475129efdf4355572ad38d80a2fa44ef88e7517d64641572585c995d"}
Nov 02 15:30:01.774692 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.774666    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg" event={"ID":"9a174897-47d1-4a4c-994a-7186e7e405ef","Type":"ContainerStarted","Data":"d46d676c64499d40c26afb20eb8542699ad207cd8e31695aa40e8be389331a33"}
Nov 02 15:30:01.775904 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.775881    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7" event={"ID":"585afb52-1760-4ed3-8fbb-e0e5e3077372","Type":"ContainerStarted","Data":"536e05acb6fd9736f694805077809fc1174ec76f759ce6b5d39b12e03f0c2371"}
Nov 02 15:30:01.777180 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.777158    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb" event={"ID":"2ae86072-ec09-40f3-93f1-452f118684ba","Type":"ContainerStarted","Data":"1439e23ae5733c8f76d0f96f0c36f13428e1c75bf21cb2d0148aca995e39dfe0"}
Nov 02 15:30:01.778849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.778823    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6" event={"ID":"f33e9cd6-6338-4897-9adc-8282a614e231","Type":"ContainerStarted","Data":"10d5fd14feeed689e8c1a97523b4559f0bd2db6e1ac328e94cb0e1667a8dd2e2"}
Nov 02 15:30:01.778961 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.778871    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6" event={"ID":"f33e9cd6-6338-4897-9adc-8282a614e231","Type":"ContainerStarted","Data":"af8ea3c1c889f3abfe967f5eb86dff327c4092414acaa60f37be66a1e146a417"}
Nov 02 15:30:01.780369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.780347    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6" event={"ID":"13461dec-71c7-4ffc-aabb-f84f820c4502","Type":"ContainerStarted","Data":"ca59401fa87c0ee4249109a7cef8f3f43fe6b7ee56ff2d2c2a9ec7d191d22c24"}
Nov 02 15:30:01.781646 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.781622    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd" event={"ID":"9fba93da-7b0c-4456-8c69-461cfbb3fca9","Type":"ContainerStarted","Data":"ec6fde9f396923663662372e785dcd55b90643d3881e9313419934b527d7c7bb"}
Nov 02 15:30:01.961770 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.961662    4891 scope.go:122] "RemoveContainer" containerID="126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996"
Nov 02 15:30:01.962173 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:01.962148    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996\": container with ID starting with 126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996 not found: ID does not exist" containerID="126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996"
Nov 02 15:30:01.962252 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:01.962183    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996"} err="failed to get container status \"126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996\": rpc error: code = NotFound desc = could not find container \"126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996\": container with ID starting with 126261fd7630a3f570cf4c5d68a6b4cb1d58b6a633fcc93a315ec9e29f48c996 not found: ID does not exist"
Nov 02 15:30:02.006665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.006611    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6" podStartSLOduration=2.006598048 podStartE2EDuration="2.006598048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.002663347 +0000 UTC m=+2009.595280147" watchObservedRunningTime="2026-11-02 15:30:02.006598048 +0000 UTC m=+2009.599214846"
Nov 02 15:30:02.185475 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.185439    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"]
Nov 02 15:30:02.190795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.190769    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-7shxl"]
Nov 02 15:30:02.253254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.253214    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"]
Nov 02 15:30:02.308669 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.308616    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb" podStartSLOduration=2.308601263 podStartE2EDuration="2.308601263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.305119496 +0000 UTC m=+2009.897736296" watchObservedRunningTime="2026-11-02 15:30:02.308601263 +0000 UTC m=+2009.901218063"
Nov 02 15:30:02.357844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.357813    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"]
Nov 02 15:30:02.421372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.421226    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" containerID="cri-o://c7eefc33ae18808adad5b5c9c2f7963d85539e2d1208d2ad7a472118a02730a7" gracePeriod=45
Nov 02 15:30:02.475587 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.475543    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" containerID="cri-o://dc8c26b7bb9d3458a60f230790d4bbf37e349c7f763e2ab8cc3b73600dbe2460" gracePeriod=45
Nov 02 15:30:02.513382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.513339    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": dial tcp 10.132.0.55:8000: connect: connection refused"
Nov 02 15:30:02.524276 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.524245    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" path="/var/lib/kubelet/pods/47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5/volumes"
Nov 02 15:30:02.719031 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.718949    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": dial tcp 10.132.0.91:8000: connect: connection refused"
Nov 02 15:30:02.802635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.801140    4891 generic.go:356] "Generic (PLEG): container finished" podID="96c01d26-e151-44a6-8501-d9618ad61ab9" containerID="99ea4324f6a3d548221e4c02941c4f7b8d822d6633801f1bff254eedba638a5f" exitCode=60
Nov 02 15:30:02.802635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.801259    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9" event={"ID":"96c01d26-e151-44a6-8501-d9618ad61ab9","Type":"ContainerDied","Data":"99ea4324f6a3d548221e4c02941c4f7b8d822d6633801f1bff254eedba638a5f"}
Nov 02 15:30:02.802635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.801844    4891 scope.go:122] "RemoveContainer" containerID="99ea4324f6a3d548221e4c02941c4f7b8d822d6633801f1bff254eedba638a5f"
Nov 02 15:30:02.815425 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.815388    4891 generic.go:356] "Generic (PLEG): container finished" podID="84777af8-4c53-4f7e-9a1a-940e6798353e" containerID="522937bcdb07377a02b1fee17e0a3b57bab0ce13bc75ffaf800adfdb03d4c872" exitCode=60
Nov 02 15:30:02.819101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.815560    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb" event={"ID":"84777af8-4c53-4f7e-9a1a-940e6798353e","Type":"ContainerDied","Data":"522937bcdb07377a02b1fee17e0a3b57bab0ce13bc75ffaf800adfdb03d4c872"}
Nov 02 15:30:02.819332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.815807    4891 scope.go:122] "RemoveContainer" containerID="522937bcdb07377a02b1fee17e0a3b57bab0ce13bc75ffaf800adfdb03d4c872"
Nov 02 15:30:02.822535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.822374    4891 generic.go:356] "Generic (PLEG): container finished" podID="59ca96d6-a378-43b9-855f-29b40a217715" containerID="c7eefc33ae18808adad5b5c9c2f7963d85539e2d1208d2ad7a472118a02730a7" exitCode=0
Nov 02 15:30:02.822535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.822495    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" event={"ID":"59ca96d6-a378-43b9-855f-29b40a217715","Type":"ContainerDied","Data":"c7eefc33ae18808adad5b5c9c2f7963d85539e2d1208d2ad7a472118a02730a7"}
Nov 02 15:30:02.824667 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.824640    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb" event={"ID":"e212aa5a-ae15-4ac4-8958-d38c83b2658f","Type":"ContainerStarted","Data":"7513a3dff4c5ceb9d1695ed9e805f5e6f82eba94ae6eef9b5295ace0d8291b7f"}
Nov 02 15:30:02.836572 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.836532    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9" event={"ID":"db707186-6796-4dad-893d-cb8ced999821","Type":"ContainerStarted","Data":"345691fa36bf31bd6acaf2e4045740bfadcde2e9668f5cfc619d825ea10ae8a9"}
Nov 02 15:30:02.836817 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.836797    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9" event={"ID":"db707186-6796-4dad-893d-cb8ced999821","Type":"ContainerStarted","Data":"b9d50b5f983af2caca29e4c1229b67acbd194e29cdc3816d7df4b383e28c055b"}
Nov 02 15:30:02.850151 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.848567    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls" event={"ID":"de5a0c88-2c43-48b1-b966-edb2603ef847","Type":"ContainerStarted","Data":"add856e6f9b51d1b4af687d405365d26979ff63fc9b62aac6f6eed0227f263df"}
Nov 02 15:30:02.875589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.875549    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg" event={"ID":"9a174897-47d1-4a4c-994a-7186e7e405ef","Type":"ContainerStarted","Data":"b4c775cfe1206f5fa249b7dbad102ce206ddc1282039f6f16bbceffc9d5852f6"}
Nov 02 15:30:02.887162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.887127    4891 generic.go:356] "Generic (PLEG): container finished" podID="585afb52-1760-4ed3-8fbb-e0e5e3077372" containerID="a6341b3ec28d86512b22bca2c96743a4ad401c9e2006aaf20b9c266b33c257c4" exitCode=60
Nov 02 15:30:02.888510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.888487    4891 scope.go:122] "RemoveContainer" containerID="a6341b3ec28d86512b22bca2c96743a4ad401c9e2006aaf20b9c266b33c257c4"
Nov 02 15:30:02.889220 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.889196    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7" event={"ID":"585afb52-1760-4ed3-8fbb-e0e5e3077372","Type":"ContainerDied","Data":"a6341b3ec28d86512b22bca2c96743a4ad401c9e2006aaf20b9c266b33c257c4"}
Nov 02 15:30:02.892868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.892842    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb" event={"ID":"2ae86072-ec09-40f3-93f1-452f118684ba","Type":"ContainerStarted","Data":"e55504ea49f4481473013dd8d74c015e94a02e0dd7e6cadd373fbaa41576b7cd"}
Nov 02 15:30:02.895637 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.895607    4891 generic.go:356] "Generic (PLEG): container finished" podID="f33e9cd6-6338-4897-9adc-8282a614e231" containerID="10d5fd14feeed689e8c1a97523b4559f0bd2db6e1ac328e94cb0e1667a8dd2e2" exitCode=60
Nov 02 15:30:02.896115 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.896095    4891 scope.go:122] "RemoveContainer" containerID="10d5fd14feeed689e8c1a97523b4559f0bd2db6e1ac328e94cb0e1667a8dd2e2"
Nov 02 15:30:02.896420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.896395    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6" event={"ID":"f33e9cd6-6338-4897-9adc-8282a614e231","Type":"ContainerDied","Data":"10d5fd14feeed689e8c1a97523b4559f0bd2db6e1ac328e94cb0e1667a8dd2e2"}
Nov 02 15:30:02.896541 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.896189    4891 patch_prober.go:29] interesting pod/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:30:02.896629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.896617    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" podUID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:30:02.898033 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.898008    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6" event={"ID":"13461dec-71c7-4ffc-aabb-f84f820c4502","Type":"ContainerStarted","Data":"e4650ca9f14ac73b35de7075f1e39e77426a9fb23e683e92550b00ddf3f94e1d"}
Nov 02 15:30:02.899818 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.899794    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd" event={"ID":"9fba93da-7b0c-4456-8c69-461cfbb3fca9","Type":"ContainerStarted","Data":"2ef7fefa1b3dbb4bb05474165d6d5ebaf4a4b937bc9d52be00bf0d5474f32528"}
Nov 02 15:30:02.902530 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.902505    4891 generic.go:356] "Generic (PLEG): container finished" podID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerID="dc8c26b7bb9d3458a60f230790d4bbf37e349c7f763e2ab8cc3b73600dbe2460" exitCode=0
Nov 02 15:30:02.902623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.902585    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" event={"ID":"9e14887d-6b64-456e-ad0b-3e0b34789cbe","Type":"ContainerDied","Data":"dc8c26b7bb9d3458a60f230790d4bbf37e349c7f763e2ab8cc3b73600dbe2460"}
Nov 02 15:30:02.904567 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.904549    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp" event={"ID":"d59415e5-4268-42af-b993-3eea7e047ef5","Type":"ContainerStarted","Data":"55b5f5951959a0ad4b68a48a7909e1a45bcf0aa77fdc494ad4d78befe2fd92d9"}
Nov 02 15:30:02.970410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.970264    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:30:02.988748 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:02.988722    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:30:03.048222 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.044273    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg" podStartSLOduration=3.044259273 podStartE2EDuration="3.044259273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.040771148 +0000 UTC m=+2010.633387948" watchObservedRunningTime="2026-11-02 15:30:03.044259273 +0000 UTC m=+2010.636876066"
Nov 02 15:30:03.052998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.052960    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-dshm\") pod \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") "
Nov 02 15:30:03.053403 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.053277    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tls-certs\") pod \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") "
Nov 02 15:30:03.053403 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.053357    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-model-cache\") pod \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") "
Nov 02 15:30:03.053570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.053404    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-tmp-dir\") pod \"59ca96d6-a378-43b9-855f-29b40a217715\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") "
Nov 02 15:30:03.053570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.053469    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59ca96d6-a378-43b9-855f-29b40a217715-tls-certs\" (UniqueName: \"kubernetes.io/secret/59ca96d6-a378-43b9-855f-29b40a217715-tls-certs\") pod \"59ca96d6-a378-43b9-855f-29b40a217715\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") "
Nov 02 15:30:03.053681 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.053555    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-home\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-home\") pod \"59ca96d6-a378-43b9-855f-29b40a217715\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") "
Nov 02 15:30:03.053875 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.053843    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-model-cache" pod "9e14887d-6b64-456e-ad0b-3e0b34789cbe" (UID: "9e14887d-6b64-456e-ad0b-3e0b34789cbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:03.053984 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.053952    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-tmp-dir" pod "59ca96d6-a378-43b9-855f-29b40a217715" (UID: "59ca96d6-a378-43b9-855f-29b40a217715"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:03.054350 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.054280    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-home" pod "59ca96d6-a378-43b9-855f-29b40a217715" (UID: "59ca96d6-a378-43b9-855f-29b40a217715"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:03.054451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.054285    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tmp-dir\") pod \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") "
Nov 02 15:30:03.054513 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.054492    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-model-cache\") pod \"59ca96d6-a378-43b9-855f-29b40a217715\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") "
Nov 02 15:30:03.054569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.054514    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tmp-dir" pod "9e14887d-6b64-456e-ad0b-3e0b34789cbe" (UID: "9e14887d-6b64-456e-ad0b-3e0b34789cbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:03.054569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.054554    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ca96d6-a378-43b9-855f-29b40a217715-kube-api-access-p75bl\" (UniqueName: \"kubernetes.io/projected/59ca96d6-a378-43b9-855f-29b40a217715-kube-api-access-p75bl\") pod \"59ca96d6-a378-43b9-855f-29b40a217715\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") "
Nov 02 15:30:03.054671 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.054627    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-home\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-home\") pod \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") "
Nov 02 15:30:03.054671 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.054665    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e14887d-6b64-456e-ad0b-3e0b34789cbe-kube-api-access-8zc7v\" (UniqueName: \"kubernetes.io/projected/9e14887d-6b64-456e-ad0b-3e0b34789cbe-kube-api-access-8zc7v\") pod \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\" (UID: \"9e14887d-6b64-456e-ad0b-3e0b34789cbe\") "
Nov 02 15:30:03.054793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.054690    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-dshm\") pod \"59ca96d6-a378-43b9-855f-29b40a217715\" (UID: \"59ca96d6-a378-43b9-855f-29b40a217715\") "
Nov 02 15:30:03.054793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.054711    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-model-cache" pod "59ca96d6-a378-43b9-855f-29b40a217715" (UID: "59ca96d6-a378-43b9-855f-29b40a217715"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:03.054925 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.054902    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-home" pod "9e14887d-6b64-456e-ad0b-3e0b34789cbe" (UID: "9e14887d-6b64-456e-ad0b-3e0b34789cbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:03.055773 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.055749    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:03.055934 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.055916    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:03.056009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.055936    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:03.056009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.055950    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:03.056009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.055962    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:03.056009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.055975    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:03.163852 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.163802    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb" podStartSLOduration=3.163787166 podStartE2EDuration="3.163787166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.159559719 +0000 UTC m=+2010.752176517" watchObservedRunningTime="2026-11-02 15:30:03.163787166 +0000 UTC m=+2010.756403983"
Nov 02 15:30:03.245555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.245438    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-dshm" pod "9e14887d-6b64-456e-ad0b-3e0b34789cbe" (UID: "9e14887d-6b64-456e-ad0b-3e0b34789cbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:03.245555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.245444    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ca96d6-a378-43b9-855f-29b40a217715-tls-certs" pod "59ca96d6-a378-43b9-855f-29b40a217715" (UID: "59ca96d6-a378-43b9-855f-29b40a217715"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:30:03.246259 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.245895    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ca96d6-a378-43b9-855f-29b40a217715-kube-api-access-p75bl" pod "59ca96d6-a378-43b9-855f-29b40a217715" (UID: "59ca96d6-a378-43b9-855f-29b40a217715"). InnerVolumeSpecName "kube-api-access-p75bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:03.246259 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.246185    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e14887d-6b64-456e-ad0b-3e0b34789cbe-kube-api-access-8zc7v" pod "9e14887d-6b64-456e-ad0b-3e0b34789cbe" (UID: "9e14887d-6b64-456e-ad0b-3e0b34789cbe"). InnerVolumeSpecName "kube-api-access-8zc7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:03.247602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.247485    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tls-certs" pod "9e14887d-6b64-456e-ad0b-3e0b34789cbe" (UID: "9e14887d-6b64-456e-ad0b-3e0b34789cbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:30:03.249161 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.248391    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-dshm" pod "59ca96d6-a378-43b9-855f-29b40a217715" (UID: "59ca96d6-a378-43b9-855f-29b40a217715"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:03.258651 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.258611    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zc7v\" (UniqueName: \"kubernetes.io/projected/9e14887d-6b64-456e-ad0b-3e0b34789cbe-kube-api-access-8zc7v\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:03.258651 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.258650    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ca96d6-a378-43b9-855f-29b40a217715-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:03.258828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.258665    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e14887d-6b64-456e-ad0b-3e0b34789cbe-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:03.258828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.258676    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e14887d-6b64-456e-ad0b-3e0b34789cbe-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:03.258828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.258688    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ca96d6-a378-43b9-855f-29b40a217715-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:03.258828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.258710    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p75bl\" (UniqueName: \"kubernetes.io/projected/59ca96d6-a378-43b9-855f-29b40a217715-kube-api-access-p75bl\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:03.277628 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.277572    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp" podStartSLOduration=3.277557188 podStartE2EDuration="3.277557188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.276050129 +0000 UTC m=+2010.868666931" watchObservedRunningTime="2026-11-02 15:30:03.277557188 +0000 UTC m=+2010.870173988"
Nov 02 15:30:03.507915 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.507750    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls" podStartSLOduration=3.507735991 podStartE2EDuration="3.507735991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.507270264 +0000 UTC m=+2011.099887066" watchObservedRunningTime="2026-11-02 15:30:03.507735991 +0000 UTC m=+2011.100352791"
Nov 02 15:30:03.745638 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.745578    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6" podStartSLOduration=3.745557006 podStartE2EDuration="3.745557006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.741246261 +0000 UTC m=+2011.333863071" watchObservedRunningTime="2026-11-02 15:30:03.745557006 +0000 UTC m=+2011.338173808"
Nov 02 15:30:03.858908 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.858767    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9" podStartSLOduration=3.858749879 podStartE2EDuration="3.858749879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.857837384 +0000 UTC m=+2011.450454183" watchObservedRunningTime="2026-11-02 15:30:03.858749879 +0000 UTC m=+2011.451366681"
Nov 02 15:30:03.916579 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.916545    4891 generic.go:356] "Generic (PLEG): container finished" podID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" containerID="7513a3dff4c5ceb9d1695ed9e805f5e6f82eba94ae6eef9b5295ace0d8291b7f" exitCode=60
Nov 02 15:30:03.916741 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.916628    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb" event={"ID":"e212aa5a-ae15-4ac4-8958-d38c83b2658f","Type":"ContainerDied","Data":"7513a3dff4c5ceb9d1695ed9e805f5e6f82eba94ae6eef9b5295ace0d8291b7f"}
Nov 02 15:30:03.917034 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.917006    4891 scope.go:122] "RemoveContainer" containerID="7513a3dff4c5ceb9d1695ed9e805f5e6f82eba94ae6eef9b5295ace0d8291b7f"
Nov 02 15:30:03.921607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.921579    4891 generic.go:356] "Generic (PLEG): container finished" podID="db707186-6796-4dad-893d-cb8ced999821" containerID="345691fa36bf31bd6acaf2e4045740bfadcde2e9668f5cfc619d825ea10ae8a9" exitCode=60
Nov 02 15:30:03.922093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.922075    4891 scope.go:122] "RemoveContainer" containerID="345691fa36bf31bd6acaf2e4045740bfadcde2e9668f5cfc619d825ea10ae8a9"
Nov 02 15:30:03.922562 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.922509    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9" event={"ID":"db707186-6796-4dad-893d-cb8ced999821","Type":"ContainerDied","Data":"345691fa36bf31bd6acaf2e4045740bfadcde2e9668f5cfc619d825ea10ae8a9"}
Nov 02 15:30:03.925714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.924661    4891 generic.go:356] "Generic (PLEG): container finished" podID="de5a0c88-2c43-48b1-b966-edb2603ef847" containerID="add856e6f9b51d1b4af687d405365d26979ff63fc9b62aac6f6eed0227f263df" exitCode=60
Nov 02 15:30:03.925714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.924776    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls" event={"ID":"de5a0c88-2c43-48b1-b966-edb2603ef847","Type":"ContainerDied","Data":"add856e6f9b51d1b4af687d405365d26979ff63fc9b62aac6f6eed0227f263df"}
Nov 02 15:30:03.925714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.925156    4891 scope.go:122] "RemoveContainer" containerID="add856e6f9b51d1b4af687d405365d26979ff63fc9b62aac6f6eed0227f263df"
Nov 02 15:30:03.929755 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.929733    4891 generic.go:356] "Generic (PLEG): container finished" podID="9a174897-47d1-4a4c-994a-7186e7e405ef" containerID="b4c775cfe1206f5fa249b7dbad102ce206ddc1282039f6f16bbceffc9d5852f6" exitCode=60
Nov 02 15:30:03.929868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.929831    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg" event={"ID":"9a174897-47d1-4a4c-994a-7186e7e405ef","Type":"ContainerDied","Data":"b4c775cfe1206f5fa249b7dbad102ce206ddc1282039f6f16bbceffc9d5852f6"}
Nov 02 15:30:03.930733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.930715    4891 scope.go:122] "RemoveContainer" containerID="b4c775cfe1206f5fa249b7dbad102ce206ddc1282039f6f16bbceffc9d5852f6"
Nov 02 15:30:03.932491 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.932469    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7" event={"ID":"585afb52-1760-4ed3-8fbb-e0e5e3077372","Type":"ContainerStarted","Data":"ce1d8dd68328efbf31370b799c94604bf2d348dfdd69791158de61a29a2e00c7"}
Nov 02 15:30:03.941987 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.941963    4891 generic.go:356] "Generic (PLEG): container finished" podID="2ae86072-ec09-40f3-93f1-452f118684ba" containerID="e55504ea49f4481473013dd8d74c015e94a02e0dd7e6cadd373fbaa41576b7cd" exitCode=60
Nov 02 15:30:03.942141 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.942093    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb" event={"ID":"2ae86072-ec09-40f3-93f1-452f118684ba","Type":"ContainerDied","Data":"e55504ea49f4481473013dd8d74c015e94a02e0dd7e6cadd373fbaa41576b7cd"}
Nov 02 15:30:03.942822 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.942801    4891 scope.go:122] "RemoveContainer" containerID="e55504ea49f4481473013dd8d74c015e94a02e0dd7e6cadd373fbaa41576b7cd"
Nov 02 15:30:03.956129 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.956087    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6" event={"ID":"f33e9cd6-6338-4897-9adc-8282a614e231","Type":"ContainerStarted","Data":"73d8c75b5f6c9375cb2ee91496379168b1669930bea73a64d1285e2dc33c7dca"}
Nov 02 15:30:03.958846 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.958816    4891 generic.go:356] "Generic (PLEG): container finished" podID="13461dec-71c7-4ffc-aabb-f84f820c4502" containerID="e4650ca9f14ac73b35de7075f1e39e77426a9fb23e683e92550b00ddf3f94e1d" exitCode=60
Nov 02 15:30:03.958969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.958909    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6" event={"ID":"13461dec-71c7-4ffc-aabb-f84f820c4502","Type":"ContainerDied","Data":"e4650ca9f14ac73b35de7075f1e39e77426a9fb23e683e92550b00ddf3f94e1d"}
Nov 02 15:30:03.959283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.959252    4891 scope.go:122] "RemoveContainer" containerID="e4650ca9f14ac73b35de7075f1e39e77426a9fb23e683e92550b00ddf3f94e1d"
Nov 02 15:30:03.961278 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.961258    4891 generic.go:356] "Generic (PLEG): container finished" podID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" containerID="2ef7fefa1b3dbb4bb05474165d6d5ebaf4a4b937bc9d52be00bf0d5474f32528" exitCode=60
Nov 02 15:30:03.961657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.961641    4891 scope.go:122] "RemoveContainer" containerID="2ef7fefa1b3dbb4bb05474165d6d5ebaf4a4b937bc9d52be00bf0d5474f32528"
Nov 02 15:30:03.962020 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.961998    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd" event={"ID":"9fba93da-7b0c-4456-8c69-461cfbb3fca9","Type":"ContainerDied","Data":"2ef7fefa1b3dbb4bb05474165d6d5ebaf4a4b937bc9d52be00bf0d5474f32528"}
Nov 02 15:30:03.969410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.965364    4891 generic.go:356] "Generic (PLEG): container finished" podID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" containerID="8196030bf19326e58de26e793a3a2f39c145d0de5b0fcf900a81a7e048e927b6" exitCode=0
Nov 02 15:30:03.969410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.965467    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" event={"ID":"9bee6b2b-9e64-4b60-9f35-0775382a60f1","Type":"ContainerDied","Data":"8196030bf19326e58de26e793a3a2f39c145d0de5b0fcf900a81a7e048e927b6"}
Nov 02 15:30:03.969410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.968847    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5" event={"ID":"9e14887d-6b64-456e-ad0b-3e0b34789cbe","Type":"ContainerDied","Data":"fb83516a0724a426a99fd4dfbbe98298226a54999750879b81cdd31ca88c1b93"}
Nov 02 15:30:03.969410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.968887    4891 scope.go:122] "RemoveContainer" containerID="dc8c26b7bb9d3458a60f230790d4bbf37e349c7f763e2ab8cc3b73600dbe2460"
Nov 02 15:30:03.969410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.969083    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"
Nov 02 15:30:03.989497 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.988639    4891 generic.go:356] "Generic (PLEG): container finished" podID="d59415e5-4268-42af-b993-3eea7e047ef5" containerID="55b5f5951959a0ad4b68a48a7909e1a45bcf0aa77fdc494ad4d78befe2fd92d9" exitCode=60
Nov 02 15:30:03.989497 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.988764    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp" event={"ID":"d59415e5-4268-42af-b993-3eea7e047ef5","Type":"ContainerDied","Data":"55b5f5951959a0ad4b68a48a7909e1a45bcf0aa77fdc494ad4d78befe2fd92d9"}
Nov 02 15:30:03.989497 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.989038    4891 scope.go:122] "RemoveContainer" containerID="55b5f5951959a0ad4b68a48a7909e1a45bcf0aa77fdc494ad4d78befe2fd92d9"
Nov 02 15:30:03.999135 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:03.998487    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9" event={"ID":"96c01d26-e151-44a6-8501-d9618ad61ab9","Type":"ContainerStarted","Data":"e9d6b034923f1f36e415635b7a3537ec78690a6067a778903f319acd8129c404"}
Nov 02 15:30:04.032569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.031288    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb" event={"ID":"84777af8-4c53-4f7e-9a1a-940e6798353e","Type":"ContainerStarted","Data":"31cb1eea77d9bc4583b39d7e316d2f774f53783fa645d1689564ca111ba63a00"}
Nov 02 15:30:04.065490 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.065342    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"
Nov 02 15:30:04.065588 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.065260    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26" event={"ID":"59ca96d6-a378-43b9-855f-29b40a217715","Type":"ContainerDied","Data":"5ed0bb23bc6e7ca9a71a928b4c44af4bc807abd57a62949cc85b7b0c0902e961"}
Nov 02 15:30:04.065725 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.065710    4891 scope.go:122] "RemoveContainer" containerID="c7eefc33ae18808adad5b5c9c2f7963d85539e2d1208d2ad7a472118a02730a7"
Nov 02 15:30:04.095190 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.094989    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:30:04.106683 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.104715    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb" podStartSLOduration=4.104695186 podStartE2EDuration="4.104695186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:04.102941371 +0000 UTC m=+2011.695558170" watchObservedRunningTime="2026-11-02 15:30:04.104695186 +0000 UTC m=+2011.697311986"
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.194894    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istiod-ca-cert\") pod \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") "
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.194981    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-socket\") pod \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") "
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.195067    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-kube-api-access-7t472\" (UniqueName: \"kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-kube-api-access-7t472\") pod \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") "
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.195115    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-data\") pod \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") "
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.195177    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-credential-socket\") pod \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") "
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.195208    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-certs\") pod \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") "
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.195275    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-podinfo\") pod \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") "
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.195372    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-envoy\") pod \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") "
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.195410    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-token\" (UniqueName: \"kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-token\") pod \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\" (UID: \"9bee6b2b-9e64-4b60-9f35-0775382a60f1\") "
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.195610    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-data" pod "9bee6b2b-9e64-4b60-9f35-0775382a60f1" (UID: "9bee6b2b-9e64-4b60-9f35-0775382a60f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.195964    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istiod-ca-cert" pod "9bee6b2b-9e64-4b60-9f35-0775382a60f1" (UID: "9bee6b2b-9e64-4b60-9f35-0775382a60f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.196178    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.196199    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.196284    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-socket" pod "9bee6b2b-9e64-4b60-9f35-0775382a60f1" (UID: "9bee6b2b-9e64-4b60-9f35-0775382a60f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.196551    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-certs" pod "9bee6b2b-9e64-4b60-9f35-0775382a60f1" (UID: "9bee6b2b-9e64-4b60-9f35-0775382a60f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:04.199527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.196787    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-credential-socket" pod "9bee6b2b-9e64-4b60-9f35-0775382a60f1" (UID: "9bee6b2b-9e64-4b60-9f35-0775382a60f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:04.230189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.230133    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd" podStartSLOduration=4.230116629 podStartE2EDuration="4.230116629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:04.225453197 +0000 UTC m=+2011.818069997" watchObservedRunningTime="2026-11-02 15:30:04.230116629 +0000 UTC m=+2011.822733429"
Nov 02 15:30:04.253205 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.253167    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-kube-api-access-7t472" pod "9bee6b2b-9e64-4b60-9f35-0775382a60f1" (UID: "9bee6b2b-9e64-4b60-9f35-0775382a60f1"). InnerVolumeSpecName "kube-api-access-7t472". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:04.253742 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.253718    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-podinfo" pod "9bee6b2b-9e64-4b60-9f35-0775382a60f1" (UID: "9bee6b2b-9e64-4b60-9f35-0775382a60f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:30:04.253830 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.253725    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-token" pod "9bee6b2b-9e64-4b60-9f35-0775382a60f1" (UID: "9bee6b2b-9e64-4b60-9f35-0775382a60f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:04.254030 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.254001    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-envoy" pod "9bee6b2b-9e64-4b60-9f35-0775382a60f1" (UID: "9bee6b2b-9e64-4b60-9f35-0775382a60f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:04.297355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.297256    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:04.297355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.297284    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t472\" (UniqueName: \"kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-kube-api-access-7t472\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:04.297355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.297298    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:04.297355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.297341    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:04.297355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.297356    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:04.297628 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.297369    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:04.297628 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.297384    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bee6b2b-9e64-4b60-9f35-0775382a60f1-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:04.559070 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.559000    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"]
Nov 02 15:30:04.559070 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.559049    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/distinct-2-w0-ba49d0-kserve-cfd9978c6-q7r26"]
Nov 02 15:30:04.950183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.950145    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"]
Nov 02 15:30:04.957220 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:04.957188    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ba49d0/simulator-w0-ba49d0-kserve-647dc699b9-s9mc5"]
Nov 02 15:30:05.076803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.076753    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk" event={"ID":"9bee6b2b-9e64-4b60-9f35-0775382a60f1","Type":"ContainerDied","Data":"65fc26e3b088c7471b17aee4b2eeca5747691b38c31ac3a15418e3c534a889c9"}
Nov 02 15:30:05.077074 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.076771    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"
Nov 02 15:30:05.077169 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.077058    4891 scope.go:122] "RemoveContainer" containerID="8196030bf19326e58de26e793a3a2f39c145d0de5b0fcf900a81a7e048e927b6"
Nov 02 15:30:05.080833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.080802    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp" event={"ID":"d59415e5-4268-42af-b993-3eea7e047ef5","Type":"ContainerStarted","Data":"562886e777c691808b9124c60457aed6b63c54b1c8a0dbf5768e5d92f9d84bc7"}
Nov 02 15:30:05.082985 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.082959    4891 generic.go:356] "Generic (PLEG): container finished" podID="96c01d26-e151-44a6-8501-d9618ad61ab9" containerID="e9d6b034923f1f36e415635b7a3537ec78690a6067a778903f319acd8129c404" exitCode=60
Nov 02 15:30:05.083112 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.083041    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9" event={"ID":"96c01d26-e151-44a6-8501-d9618ad61ab9","Type":"ContainerDied","Data":"e9d6b034923f1f36e415635b7a3537ec78690a6067a778903f319acd8129c404"}
Nov 02 15:30:05.085299 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.085276    4891 generic.go:356] "Generic (PLEG): container finished" podID="84777af8-4c53-4f7e-9a1a-940e6798353e" containerID="31cb1eea77d9bc4583b39d7e316d2f774f53783fa645d1689564ca111ba63a00" exitCode=60
Nov 02 15:30:05.085432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.085338    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb" event={"ID":"84777af8-4c53-4f7e-9a1a-940e6798353e","Type":"ContainerDied","Data":"31cb1eea77d9bc4583b39d7e316d2f774f53783fa645d1689564ca111ba63a00"}
Nov 02 15:30:05.085633 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.085619    4891 scope.go:122] "RemoveContainer" containerID="31cb1eea77d9bc4583b39d7e316d2f774f53783fa645d1689564ca111ba63a00"
Nov 02 15:30:05.085944 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:05.085926    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb_odh-ai-gateway-infra(84777af8-4c53-4f7e-9a1a-940e6798353e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e"
Nov 02 15:30:05.088522 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.088494    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb" event={"ID":"e212aa5a-ae15-4ac4-8958-d38c83b2658f","Type":"ContainerStarted","Data":"5e0800b902c8904148af9ba0baa09621838a27da0bab3899a4962cc4e547f467"}
Nov 02 15:30:05.090438 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.090412    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9" event={"ID":"db707186-6796-4dad-893d-cb8ced999821","Type":"ContainerStarted","Data":"c7c4aa825a60297e73d1c46ae6975268a1c2d4b1561cecedc5d8c91588544907"}
Nov 02 15:30:05.090695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.090678    4891 scope.go:122] "RemoveContainer" containerID="e9d6b034923f1f36e415635b7a3537ec78690a6067a778903f319acd8129c404"
Nov 02 15:30:05.090760 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.090723    4891 scope.go:122] "RemoveContainer" containerID="99ea4324f6a3d548221e4c02941c4f7b8d822d6633801f1bff254eedba638a5f"
Nov 02 15:30:05.090972 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:05.090952    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9_odh-ai-gateway-infra(96c01d26-e151-44a6-8501-d9618ad61ab9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9"
Nov 02 15:30:05.097101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.092638    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls" event={"ID":"de5a0c88-2c43-48b1-b966-edb2603ef847","Type":"ContainerStarted","Data":"6813c8ae65c09edb8dcb52cdfa47de281762fc029cc3ab0d7be4f38998db157c"}
Nov 02 15:30:05.100081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.100055    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg" event={"ID":"9a174897-47d1-4a4c-994a-7186e7e405ef","Type":"ContainerStarted","Data":"0d03dab745986c4edde96aa071c82d20f15cbe6221e9f5d58cd29c717b2e49d7"}
Nov 02 15:30:05.102029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.102008    4891 generic.go:356] "Generic (PLEG): container finished" podID="585afb52-1760-4ed3-8fbb-e0e5e3077372" containerID="ce1d8dd68328efbf31370b799c94604bf2d348dfdd69791158de61a29a2e00c7" exitCode=60
Nov 02 15:30:05.102117 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.102087    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7" event={"ID":"585afb52-1760-4ed3-8fbb-e0e5e3077372","Type":"ContainerDied","Data":"ce1d8dd68328efbf31370b799c94604bf2d348dfdd69791158de61a29a2e00c7"}
Nov 02 15:30:05.102319 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.102291    4891 scope.go:122] "RemoveContainer" containerID="ce1d8dd68328efbf31370b799c94604bf2d348dfdd69791158de61a29a2e00c7"
Nov 02 15:30:05.102582 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:05.102556    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7_odh-ai-gateway-infra(585afb52-1760-4ed3-8fbb-e0e5e3077372)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372"
Nov 02 15:30:05.104082 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.104002    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb" event={"ID":"2ae86072-ec09-40f3-93f1-452f118684ba","Type":"ContainerStarted","Data":"308dbac1b2e836db75332f8383dcb72b80c9865637ade38645d5aa1b10859e3a"}
Nov 02 15:30:05.106061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.106038    4891 generic.go:356] "Generic (PLEG): container finished" podID="f33e9cd6-6338-4897-9adc-8282a614e231" containerID="73d8c75b5f6c9375cb2ee91496379168b1669930bea73a64d1285e2dc33c7dca" exitCode=60
Nov 02 15:30:05.106170 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.106105    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6" event={"ID":"f33e9cd6-6338-4897-9adc-8282a614e231","Type":"ContainerDied","Data":"73d8c75b5f6c9375cb2ee91496379168b1669930bea73a64d1285e2dc33c7dca"}
Nov 02 15:30:05.106590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.106569    4891 scope.go:122] "RemoveContainer" containerID="73d8c75b5f6c9375cb2ee91496379168b1669930bea73a64d1285e2dc33c7dca"
Nov 02 15:30:05.106860 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:05.106838    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6_odh-ai-gateway-infra(f33e9cd6-6338-4897-9adc-8282a614e231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6" podUID="f33e9cd6-6338-4897-9adc-8282a614e231"
Nov 02 15:30:05.107863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.107845    4891 scope.go:122] "RemoveContainer" containerID="522937bcdb07377a02b1fee17e0a3b57bab0ce13bc75ffaf800adfdb03d4c872"
Nov 02 15:30:05.108436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.108412    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6" event={"ID":"13461dec-71c7-4ffc-aabb-f84f820c4502","Type":"ContainerStarted","Data":"922587226e4ed06b06183367b66859dc484d6ff77a151c146333c0cb26faf40c"}
Nov 02 15:30:05.110220 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.110202    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd" event={"ID":"9fba93da-7b0c-4456-8c69-461cfbb3fca9","Type":"ContainerStarted","Data":"b12e91f483de481f3953caf8b014e964d7401bad502e795a8c443ba91e5be86f"}
Nov 02 15:30:05.122225 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.122205    4891 scope.go:122] "RemoveContainer" containerID="a6341b3ec28d86512b22bca2c96743a4ad401c9e2006aaf20b9c266b33c257c4"
Nov 02 15:30:05.136225 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.136203    4891 scope.go:122] "RemoveContainer" containerID="10d5fd14feeed689e8c1a97523b4559f0bd2db6e1ac328e94cb0e1667a8dd2e2"
Nov 02 15:30:05.670796 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:05.670682    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7" podStartSLOduration=5.670667991 podStartE2EDuration="5.670667991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:05.668452674 +0000 UTC m=+2013.261069472" watchObservedRunningTime="2026-11-02 15:30:05.670667991 +0000 UTC m=+2013.263284791"
Nov 02 15:30:06.044566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.044509    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9" podStartSLOduration=6.044494869 podStartE2EDuration="6.044494869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:06.042798956 +0000 UTC m=+2013.635415758" watchObservedRunningTime="2026-11-02 15:30:06.044494869 +0000 UTC m=+2013.637111679"
Nov 02 15:30:06.120615 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.120578    4891 generic.go:356] "Generic (PLEG): container finished" podID="9a174897-47d1-4a4c-994a-7186e7e405ef" containerID="0d03dab745986c4edde96aa071c82d20f15cbe6221e9f5d58cd29c717b2e49d7" exitCode=60
Nov 02 15:30:06.120799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.120664    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg" event={"ID":"9a174897-47d1-4a4c-994a-7186e7e405ef","Type":"ContainerDied","Data":"0d03dab745986c4edde96aa071c82d20f15cbe6221e9f5d58cd29c717b2e49d7"}
Nov 02 15:30:06.120799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.120735    4891 scope.go:122] "RemoveContainer" containerID="b4c775cfe1206f5fa249b7dbad102ce206ddc1282039f6f16bbceffc9d5852f6"
Nov 02 15:30:06.120996 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.120979    4891 scope.go:122] "RemoveContainer" containerID="0d03dab745986c4edde96aa071c82d20f15cbe6221e9f5d58cd29c717b2e49d7"
Nov 02 15:30:06.121422 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:06.121394    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg_odh-ai-gateway-infra(9a174897-47d1-4a4c-994a-7186e7e405ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef"
Nov 02 15:30:06.122901 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.122874    4891 scope.go:122] "RemoveContainer" containerID="ce1d8dd68328efbf31370b799c94604bf2d348dfdd69791158de61a29a2e00c7"
Nov 02 15:30:06.123135 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:06.123114    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7_odh-ai-gateway-infra(585afb52-1760-4ed3-8fbb-e0e5e3077372)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372"
Nov 02 15:30:06.124418 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.124397    4891 generic.go:356] "Generic (PLEG): container finished" podID="2ae86072-ec09-40f3-93f1-452f118684ba" containerID="308dbac1b2e836db75332f8383dcb72b80c9865637ade38645d5aa1b10859e3a" exitCode=60
Nov 02 15:30:06.124518 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.124481    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb" event={"ID":"2ae86072-ec09-40f3-93f1-452f118684ba","Type":"ContainerDied","Data":"308dbac1b2e836db75332f8383dcb72b80c9865637ade38645d5aa1b10859e3a"}
Nov 02 15:30:06.124696 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.124680    4891 scope.go:122] "RemoveContainer" containerID="308dbac1b2e836db75332f8383dcb72b80c9865637ade38645d5aa1b10859e3a"
Nov 02 15:30:06.124911 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:06.124889    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb_odh-ai-gateway-infra(2ae86072-ec09-40f3-93f1-452f118684ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb" podUID="2ae86072-ec09-40f3-93f1-452f118684ba"
Nov 02 15:30:06.126468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.126445    4891 scope.go:122] "RemoveContainer" containerID="73d8c75b5f6c9375cb2ee91496379168b1669930bea73a64d1285e2dc33c7dca"
Nov 02 15:30:06.126634 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:06.126614    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6_odh-ai-gateway-infra(f33e9cd6-6338-4897-9adc-8282a614e231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6" podUID="f33e9cd6-6338-4897-9adc-8282a614e231"
Nov 02 15:30:06.128203 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.128182    4891 generic.go:356] "Generic (PLEG): container finished" podID="13461dec-71c7-4ffc-aabb-f84f820c4502" containerID="922587226e4ed06b06183367b66859dc484d6ff77a151c146333c0cb26faf40c" exitCode=60
Nov 02 15:30:06.128298 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.128261    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6" event={"ID":"13461dec-71c7-4ffc-aabb-f84f820c4502","Type":"ContainerDied","Data":"922587226e4ed06b06183367b66859dc484d6ff77a151c146333c0cb26faf40c"}
Nov 02 15:30:06.128520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.128501    4891 scope.go:122] "RemoveContainer" containerID="922587226e4ed06b06183367b66859dc484d6ff77a151c146333c0cb26faf40c"
Nov 02 15:30:06.128762 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:06.128739    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6_odh-ai-gateway-infra(13461dec-71c7-4ffc-aabb-f84f820c4502)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502"
Nov 02 15:30:06.130175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.130153    4891 generic.go:356] "Generic (PLEG): container finished" podID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" containerID="b12e91f483de481f3953caf8b014e964d7401bad502e795a8c443ba91e5be86f" exitCode=60
Nov 02 15:30:06.130271 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.130218    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd" event={"ID":"9fba93da-7b0c-4456-8c69-461cfbb3fca9","Type":"ContainerDied","Data":"b12e91f483de481f3953caf8b014e964d7401bad502e795a8c443ba91e5be86f"}
Nov 02 15:30:06.130411 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.130395    4891 scope.go:122] "RemoveContainer" containerID="b12e91f483de481f3953caf8b014e964d7401bad502e795a8c443ba91e5be86f"
Nov 02 15:30:06.130595 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:06.130575    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd_odh-ai-gateway-infra(9fba93da-7b0c-4456-8c69-461cfbb3fca9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9"
Nov 02 15:30:06.133122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.133100    4891 generic.go:356] "Generic (PLEG): container finished" podID="d59415e5-4268-42af-b993-3eea7e047ef5" containerID="562886e777c691808b9124c60457aed6b63c54b1c8a0dbf5768e5d92f9d84bc7" exitCode=60
Nov 02 15:30:06.133220 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.133179    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp" event={"ID":"d59415e5-4268-42af-b993-3eea7e047ef5","Type":"ContainerDied","Data":"562886e777c691808b9124c60457aed6b63c54b1c8a0dbf5768e5d92f9d84bc7"}
Nov 02 15:30:06.133408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.133385    4891 scope.go:122] "RemoveContainer" containerID="562886e777c691808b9124c60457aed6b63c54b1c8a0dbf5768e5d92f9d84bc7"
Nov 02 15:30:06.133598 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:06.133578    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp_odh-ai-gateway-infra(d59415e5-4268-42af-b993-3eea7e047ef5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp" podUID="d59415e5-4268-42af-b993-3eea7e047ef5"
Nov 02 15:30:06.134901 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.134884    4891 scope.go:122] "RemoveContainer" containerID="e9d6b034923f1f36e415635b7a3537ec78690a6067a778903f319acd8129c404"
Nov 02 15:30:06.135111 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:06.135091    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9_odh-ai-gateway-infra(96c01d26-e151-44a6-8501-d9618ad61ab9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9"
Nov 02 15:30:06.136421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.136405    4891 scope.go:122] "RemoveContainer" containerID="e55504ea49f4481473013dd8d74c015e94a02e0dd7e6cadd373fbaa41576b7cd"
Nov 02 15:30:06.136544 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.136529    4891 scope.go:122] "RemoveContainer" containerID="31cb1eea77d9bc4583b39d7e316d2f774f53783fa645d1689564ca111ba63a00"
Nov 02 15:30:06.136730 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:06.136711    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb_odh-ai-gateway-infra(84777af8-4c53-4f7e-9a1a-940e6798353e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e"
Nov 02 15:30:06.138272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.138252    4891 generic.go:356] "Generic (PLEG): container finished" podID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" containerID="5e0800b902c8904148af9ba0baa09621838a27da0bab3899a4962cc4e547f467" exitCode=60
Nov 02 15:30:06.138394 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.138338    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb" event={"ID":"e212aa5a-ae15-4ac4-8958-d38c83b2658f","Type":"ContainerDied","Data":"5e0800b902c8904148af9ba0baa09621838a27da0bab3899a4962cc4e547f467"}
Nov 02 15:30:06.138524 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.138509    4891 scope.go:122] "RemoveContainer" containerID="5e0800b902c8904148af9ba0baa09621838a27da0bab3899a4962cc4e547f467"
Nov 02 15:30:06.138716 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:06.138698    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb_odh-ai-gateway-infra(e212aa5a-ae15-4ac4-8958-d38c83b2658f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f"
Nov 02 15:30:06.140106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.140087    4891 generic.go:356] "Generic (PLEG): container finished" podID="db707186-6796-4dad-893d-cb8ced999821" containerID="c7c4aa825a60297e73d1c46ae6975268a1c2d4b1561cecedc5d8c91588544907" exitCode=60
Nov 02 15:30:06.140208 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.140106    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9" event={"ID":"db707186-6796-4dad-893d-cb8ced999821","Type":"ContainerDied","Data":"c7c4aa825a60297e73d1c46ae6975268a1c2d4b1561cecedc5d8c91588544907"}
Nov 02 15:30:06.140337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.140297    4891 scope.go:122] "RemoveContainer" containerID="c7c4aa825a60297e73d1c46ae6975268a1c2d4b1561cecedc5d8c91588544907"
Nov 02 15:30:06.140504 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:06.140485    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9_odh-ai-gateway-infra(db707186-6796-4dad-893d-cb8ced999821)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9" podUID="db707186-6796-4dad-893d-cb8ced999821"
Nov 02 15:30:06.141893 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.141873    4891 generic.go:356] "Generic (PLEG): container finished" podID="de5a0c88-2c43-48b1-b966-edb2603ef847" containerID="6813c8ae65c09edb8dcb52cdfa47de281762fc029cc3ab0d7be4f38998db157c" exitCode=60
Nov 02 15:30:06.141969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.141928    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls" event={"ID":"de5a0c88-2c43-48b1-b966-edb2603ef847","Type":"ContainerDied","Data":"6813c8ae65c09edb8dcb52cdfa47de281762fc029cc3ab0d7be4f38998db157c"}
Nov 02 15:30:06.142128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.142116    4891 scope.go:122] "RemoveContainer" containerID="6813c8ae65c09edb8dcb52cdfa47de281762fc029cc3ab0d7be4f38998db157c"
Nov 02 15:30:06.142302 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:06.142287    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls_odh-ai-gateway-infra(de5a0c88-2c43-48b1-b966-edb2603ef847)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847"
Nov 02 15:30:06.150625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.150606    4891 scope.go:122] "RemoveContainer" containerID="e4650ca9f14ac73b35de7075f1e39e77426a9fb23e683e92550b00ddf3f94e1d"
Nov 02 15:30:06.169869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.169842    4891 scope.go:122] "RemoveContainer" containerID="2ef7fefa1b3dbb4bb05474165d6d5ebaf4a4b937bc9d52be00bf0d5474f32528"
Nov 02 15:30:06.187751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.187722    4891 scope.go:122] "RemoveContainer" containerID="55b5f5951959a0ad4b68a48a7909e1a45bcf0aa77fdc494ad4d78befe2fd92d9"
Nov 02 15:30:06.202008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.201987    4891 scope.go:122] "RemoveContainer" containerID="7513a3dff4c5ceb9d1695ed9e805f5e6f82eba94ae6eef9b5295ace0d8291b7f"
Nov 02 15:30:06.216019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.215997    4891 scope.go:122] "RemoveContainer" containerID="345691fa36bf31bd6acaf2e4045740bfadcde2e9668f5cfc619d825ea10ae8a9"
Nov 02 15:30:06.230227 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.230210    4891 scope.go:122] "RemoveContainer" containerID="add856e6f9b51d1b4af687d405365d26979ff63fc9b62aac6f6eed0227f263df"
Nov 02 15:30:06.529373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.529333    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ca96d6-a378-43b9-855f-29b40a217715" path="/var/lib/kubelet/pods/59ca96d6-a378-43b9-855f-29b40a217715/volumes"
Nov 02 15:30:06.530565 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:06.530536    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" path="/var/lib/kubelet/pods/9e14887d-6b64-456e-ad0b-3e0b34789cbe/volumes"
Nov 02 15:30:07.003708 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.003620    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"]
Nov 02 15:30:07.004077 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.003814    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k" podUID="7137ff72-f89d-46c8-95c7-34569d6cdd06" containerName="discovery" containerID="cri-o://157022e3fdcc0bde7cf74786aeab873e77c732bbfa49804695442136f873a0f1" gracePeriod=30
Nov 02 15:30:07.152487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.152460    4891 scope.go:122] "RemoveContainer" containerID="922587226e4ed06b06183367b66859dc484d6ff77a151c146333c0cb26faf40c"
Nov 02 15:30:07.152848 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:07.152801    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6_odh-ai-gateway-infra(13461dec-71c7-4ffc-aabb-f84f820c4502)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502"
Nov 02 15:30:07.154301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.154278    4891 scope.go:122] "RemoveContainer" containerID="b12e91f483de481f3953caf8b014e964d7401bad502e795a8c443ba91e5be86f"
Nov 02 15:30:07.154693 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:07.154665    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd_odh-ai-gateway-infra(9fba93da-7b0c-4456-8c69-461cfbb3fca9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9"
Nov 02 15:30:07.156829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.156804    4891 scope.go:122] "RemoveContainer" containerID="562886e777c691808b9124c60457aed6b63c54b1c8a0dbf5768e5d92f9d84bc7"
Nov 02 15:30:07.157135 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:07.157112    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp_odh-ai-gateway-infra(d59415e5-4268-42af-b993-3eea7e047ef5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp" podUID="d59415e5-4268-42af-b993-3eea7e047ef5"
Nov 02 15:30:07.159659 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.159633    4891 scope.go:122] "RemoveContainer" containerID="5e0800b902c8904148af9ba0baa09621838a27da0bab3899a4962cc4e547f467"
Nov 02 15:30:07.159952 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:07.159925    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb_odh-ai-gateway-infra(e212aa5a-ae15-4ac4-8958-d38c83b2658f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f"
Nov 02 15:30:07.161351 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.161328    4891 scope.go:122] "RemoveContainer" containerID="c7c4aa825a60297e73d1c46ae6975268a1c2d4b1561cecedc5d8c91588544907"
Nov 02 15:30:07.161629 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:07.161606    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9_odh-ai-gateway-infra(db707186-6796-4dad-893d-cb8ced999821)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9" podUID="db707186-6796-4dad-893d-cb8ced999821"
Nov 02 15:30:07.163943 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.163902    4891 scope.go:122] "RemoveContainer" containerID="6813c8ae65c09edb8dcb52cdfa47de281762fc029cc3ab0d7be4f38998db157c"
Nov 02 15:30:07.164237 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:07.164190    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls_odh-ai-gateway-infra(de5a0c88-2c43-48b1-b966-edb2603ef847)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847"
Nov 02 15:30:07.166197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.166175    4891 generic.go:356] "Generic (PLEG): container finished" podID="7137ff72-f89d-46c8-95c7-34569d6cdd06" containerID="157022e3fdcc0bde7cf74786aeab873e77c732bbfa49804695442136f873a0f1" exitCode=0
Nov 02 15:30:07.166282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.166256    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k" event={"ID":"7137ff72-f89d-46c8-95c7-34569d6cdd06","Type":"ContainerDied","Data":"157022e3fdcc0bde7cf74786aeab873e77c732bbfa49804695442136f873a0f1"}
Nov 02 15:30:07.168439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.168421    4891 scope.go:122] "RemoveContainer" containerID="0d03dab745986c4edde96aa071c82d20f15cbe6221e9f5d58cd29c717b2e49d7"
Nov 02 15:30:07.168661 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:07.168635    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg_odh-ai-gateway-infra(9a174897-47d1-4a4c-994a-7186e7e405ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef"
Nov 02 15:30:07.170303 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.170286    4891 scope.go:122] "RemoveContainer" containerID="308dbac1b2e836db75332f8383dcb72b80c9865637ade38645d5aa1b10859e3a"
Nov 02 15:30:07.170566 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:07.170548    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb_odh-ai-gateway-infra(2ae86072-ec09-40f3-93f1-452f118684ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb" podUID="2ae86072-ec09-40f3-93f1-452f118684ba"
Nov 02 15:30:07.253770 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.253686    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r" podUID="7390b194-bcc2-4020-98c4-73cfb8d07480" containerName="payload-processing" containerID="cri-o://84b625e3d4ddb407046a35ddbecbb18be2ef00c46abe04b81b1b2fa67fd573f3" gracePeriod=25
Nov 02 15:30:07.300060 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.300031    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:30:07.358451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.358388    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7" podUID="8ecc4598-19ce-463b-a3de-5ee6ba41a08f" containerName="payload-pre-processing" containerID="cri-o://c0fb0e54f4541765d9fad7748a1e56e1f7ff63b36422f6124fd038189948f221" gracePeriod=25
Nov 02 15:30:07.441976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.441945    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-cacerts\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-cacerts\") pod \"7137ff72-f89d-46c8-95c7-34569d6cdd06\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") "
Nov 02 15:30:07.442133 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.442075    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-kube-api-access-7skml\" (UniqueName: \"kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-kube-api-access-7skml\") pod \"7137ff72-f89d-46c8-95c7-34569d6cdd06\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") "
Nov 02 15:30:07.442133 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.442122    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-dns-cert\") pod \"7137ff72-f89d-46c8-95c7-34569d6cdd06\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") "
Nov 02 15:30:07.442260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.442151    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-ca-configmap\") pod \"7137ff72-f89d-46c8-95c7-34569d6cdd06\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") "
Nov 02 15:30:07.442260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.442173    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7137ff72-f89d-46c8-95c7-34569d6cdd06-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7137ff72-f89d-46c8-95c7-34569d6cdd06-local-certs\") pod \"7137ff72-f89d-46c8-95c7-34569d6cdd06\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") "
Nov 02 15:30:07.442260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.442255    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-token\" (UniqueName: \"kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-token\") pod \"7137ff72-f89d-46c8-95c7-34569d6cdd06\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") "
Nov 02 15:30:07.442429 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.442330    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-kubeconfig\") pod \"7137ff72-f89d-46c8-95c7-34569d6cdd06\" (UID: \"7137ff72-f89d-46c8-95c7-34569d6cdd06\") "
Nov 02 15:30:07.442679 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.442653    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-ca-configmap" pod "7137ff72-f89d-46c8-95c7-34569d6cdd06" (UID: "7137ff72-f89d-46c8-95c7-34569d6cdd06"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:07.442907 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.442886    4891 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-ca-configmap\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:07.444861 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.444797    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7137ff72-f89d-46c8-95c7-34569d6cdd06-local-certs" pod "7137ff72-f89d-46c8-95c7-34569d6cdd06" (UID: "7137ff72-f89d-46c8-95c7-34569d6cdd06"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:07.444960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.444941    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-cacerts" pod "7137ff72-f89d-46c8-95c7-34569d6cdd06" (UID: "7137ff72-f89d-46c8-95c7-34569d6cdd06"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:30:07.445022 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.444970    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-dns-cert" pod "7137ff72-f89d-46c8-95c7-34569d6cdd06" (UID: "7137ff72-f89d-46c8-95c7-34569d6cdd06"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:30:07.445022 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.444973    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-token" pod "7137ff72-f89d-46c8-95c7-34569d6cdd06" (UID: "7137ff72-f89d-46c8-95c7-34569d6cdd06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:07.445022 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.444992    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-kube-api-access-7skml" pod "7137ff72-f89d-46c8-95c7-34569d6cdd06" (UID: "7137ff72-f89d-46c8-95c7-34569d6cdd06"). InnerVolumeSpecName "kube-api-access-7skml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:07.445022 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.444995    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-kubeconfig" pod "7137ff72-f89d-46c8-95c7-34569d6cdd06" (UID: "7137ff72-f89d-46c8-95c7-34569d6cdd06"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:30:07.544262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.544231    4891 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-csr-dns-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:07.544262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.544258    4891 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7137ff72-f89d-46c8-95c7-34569d6cdd06-local-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:07.544262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.544269    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:07.544501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.544279    4891 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-istio-kubeconfig\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:07.544501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.544288    4891 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7137ff72-f89d-46c8-95c7-34569d6cdd06-cacerts\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:07.544501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.544296    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7skml\" (UniqueName: \"kubernetes.io/projected/7137ff72-f89d-46c8-95c7-34569d6cdd06-kube-api-access-7skml\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:07.725800 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.725763    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"]
Nov 02 15:30:07.730108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:07.730081    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-kzwzk"]
Nov 02 15:30:08.180436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:08.180409    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"
Nov 02 15:30:08.180837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:08.180444    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k" event={"ID":"7137ff72-f89d-46c8-95c7-34569d6cdd06","Type":"ContainerDied","Data":"5d3f9edf5a303614831b6b81feab98844e2d1b7b3b12f9d2afa557bc7c73cc50"}
Nov 02 15:30:08.180837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:08.180506    4891 scope.go:122] "RemoveContainer" containerID="157022e3fdcc0bde7cf74786aeab873e77c732bbfa49804695442136f873a0f1"
Nov 02 15:30:08.524594 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:08.524563    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" path="/var/lib/kubelet/pods/9bee6b2b-9e64-4b60-9f35-0775382a60f1/volumes"
Nov 02 15:30:08.894499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:08.894403    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"]
Nov 02 15:30:08.898413 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:08.898389    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hh82k"]
Nov 02 15:30:09.674186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:09.674144    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7" podUID="8ecc4598-19ce-463b-a3de-5ee6ba41a08f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:30:09.674597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:09.674156    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r" podUID="7390b194-bcc2-4020-98c4-73cfb8d07480" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:30:10.526209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:10.526175    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7137ff72-f89d-46c8-95c7-34569d6cdd06" path="/var/lib/kubelet/pods/7137ff72-f89d-46c8-95c7-34569d6cdd06/volumes"
Nov 02 15:30:14.274628 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:14.274544    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:30:15.518336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:15.515373    4891 scope.go:122] "RemoveContainer" containerID="5e0800b902c8904148af9ba0baa09621838a27da0bab3899a4962cc4e547f467"
Nov 02 15:30:15.518336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:15.516168    4891 scope.go:122] "RemoveContainer" containerID="73d8c75b5f6c9375cb2ee91496379168b1669930bea73a64d1285e2dc33c7dca"
Nov 02 15:30:16.253371 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:16.253272    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb" event={"ID":"e212aa5a-ae15-4ac4-8958-d38c83b2658f","Type":"ContainerStarted","Data":"39b92fc361decb16d04335247c81dc5069c8e32c15607f94ffa658ef4d4f205b"}
Nov 02 15:30:16.255467 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:16.255440    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6" event={"ID":"f33e9cd6-6338-4897-9adc-8282a614e231","Type":"ContainerStarted","Data":"d0742a106d16097c0207fe041693a67c1e5f038e3b15f8ef0a07e1058caef759"}
Nov 02 15:30:16.514994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:16.514909    4891 scope.go:122] "RemoveContainer" containerID="b12e91f483de481f3953caf8b014e964d7401bad502e795a8c443ba91e5be86f"
Nov 02 15:30:16.514994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:16.514980    4891 scope.go:122] "RemoveContainer" containerID="308dbac1b2e836db75332f8383dcb72b80c9865637ade38645d5aa1b10859e3a"
Nov 02 15:30:17.271052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.271020    4891 generic.go:356] "Generic (PLEG): container finished" podID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" containerID="39b92fc361decb16d04335247c81dc5069c8e32c15607f94ffa658ef4d4f205b" exitCode=60
Nov 02 15:30:17.271547 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.271097    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb" event={"ID":"e212aa5a-ae15-4ac4-8958-d38c83b2658f","Type":"ContainerDied","Data":"39b92fc361decb16d04335247c81dc5069c8e32c15607f94ffa658ef4d4f205b"}
Nov 02 15:30:17.271547 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.271164    4891 scope.go:122] "RemoveContainer" containerID="5e0800b902c8904148af9ba0baa09621838a27da0bab3899a4962cc4e547f467"
Nov 02 15:30:17.271666 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.271615    4891 scope.go:122] "RemoveContainer" containerID="39b92fc361decb16d04335247c81dc5069c8e32c15607f94ffa658ef4d4f205b"
Nov 02 15:30:17.272034 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:17.272000    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb_odh-ai-gateway-infra(e212aa5a-ae15-4ac4-8958-d38c83b2658f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f"
Nov 02 15:30:17.273384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.273359    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb" event={"ID":"2ae86072-ec09-40f3-93f1-452f118684ba","Type":"ContainerStarted","Data":"92ae09ce6f48795fd3008f66e7b819b4b5a0556716b88a386ce14dd4ac56184f"}
Nov 02 15:30:17.275145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.275125    4891 generic.go:356] "Generic (PLEG): container finished" podID="f33e9cd6-6338-4897-9adc-8282a614e231" containerID="d0742a106d16097c0207fe041693a67c1e5f038e3b15f8ef0a07e1058caef759" exitCode=60
Nov 02 15:30:17.275231 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.275194    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6" event={"ID":"f33e9cd6-6338-4897-9adc-8282a614e231","Type":"ContainerDied","Data":"d0742a106d16097c0207fe041693a67c1e5f038e3b15f8ef0a07e1058caef759"}
Nov 02 15:30:17.275590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.275568    4891 scope.go:122] "RemoveContainer" containerID="d0742a106d16097c0207fe041693a67c1e5f038e3b15f8ef0a07e1058caef759"
Nov 02 15:30:17.275909 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:17.275884    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6_odh-ai-gateway-infra(f33e9cd6-6338-4897-9adc-8282a614e231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6" podUID="f33e9cd6-6338-4897-9adc-8282a614e231"
Nov 02 15:30:17.277171 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.277152    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd" event={"ID":"9fba93da-7b0c-4456-8c69-461cfbb3fca9","Type":"ContainerStarted","Data":"b18e0d3441c37051afe5c63f6f91181224b1af020b0b2572a4f4132f2b81440b"}
Nov 02 15:30:17.285092 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.285078    4891 scope.go:122] "RemoveContainer" containerID="73d8c75b5f6c9375cb2ee91496379168b1669930bea73a64d1285e2dc33c7dca"
Nov 02 15:30:17.409900 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.409797    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"]
Nov 02 15:30:17.421627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.421594    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"]
Nov 02 15:30:17.457738 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.457697    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eafc0d-94wt8"]
Nov 02 15:30:17.515053 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.515019    4891 scope.go:122] "RemoveContainer" containerID="922587226e4ed06b06183367b66859dc484d6ff77a151c146333c0cb26faf40c"
Nov 02 15:30:17.515218 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.515177    4891 scope.go:122] "RemoveContainer" containerID="e9d6b034923f1f36e415635b7a3537ec78690a6067a778903f319acd8129c404"
Nov 02 15:30:17.515281 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.515266    4891 scope.go:122] "RemoveContainer" containerID="ce1d8dd68328efbf31370b799c94604bf2d348dfdd69791158de61a29a2e00c7"
Nov 02 15:30:17.515419 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.515402    4891 scope.go:122] "RemoveContainer" containerID="31cb1eea77d9bc4583b39d7e316d2f774f53783fa645d1689564ca111ba63a00"
Nov 02 15:30:17.538345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:17.538302    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"]
Nov 02 15:30:18.292167 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.292132    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb" event={"ID":"84777af8-4c53-4f7e-9a1a-940e6798353e","Type":"ContainerStarted","Data":"52a4aaa2ffefae8c5e38c63cd787447d5736ca35cc394750ff93a3d4a69c97b9"}
Nov 02 15:30:18.296066 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.296035    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7" event={"ID":"585afb52-1760-4ed3-8fbb-e0e5e3077372","Type":"ContainerStarted","Data":"db03a4f7b4991179b4df0fdd9d2804f9ed048459b65e4c614ea92d2fcbc36d17"}
Nov 02 15:30:18.298092 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.298071    4891 generic.go:356] "Generic (PLEG): container finished" podID="2ae86072-ec09-40f3-93f1-452f118684ba" containerID="92ae09ce6f48795fd3008f66e7b819b4b5a0556716b88a386ce14dd4ac56184f" exitCode=60
Nov 02 15:30:18.298200 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.298143    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb" event={"ID":"2ae86072-ec09-40f3-93f1-452f118684ba","Type":"ContainerDied","Data":"92ae09ce6f48795fd3008f66e7b819b4b5a0556716b88a386ce14dd4ac56184f"}
Nov 02 15:30:18.298200 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.298190    4891 scope.go:122] "RemoveContainer" containerID="308dbac1b2e836db75332f8383dcb72b80c9865637ade38645d5aa1b10859e3a"
Nov 02 15:30:18.298545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.298530    4891 scope.go:122] "RemoveContainer" containerID="92ae09ce6f48795fd3008f66e7b819b4b5a0556716b88a386ce14dd4ac56184f"
Nov 02 15:30:18.298854 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:18.298834    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb_odh-ai-gateway-infra(2ae86072-ec09-40f3-93f1-452f118684ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb" podUID="2ae86072-ec09-40f3-93f1-452f118684ba"
Nov 02 15:30:18.301822 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.301787    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6" event={"ID":"13461dec-71c7-4ffc-aabb-f84f820c4502","Type":"ContainerStarted","Data":"c519a79d813fdc8f35af2c47a3c6fc9a26d26307298fe2c7b6e3c9d3298a9201"}
Nov 02 15:30:18.304373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.304352    4891 generic.go:356] "Generic (PLEG): container finished" podID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" containerID="b18e0d3441c37051afe5c63f6f91181224b1af020b0b2572a4f4132f2b81440b" exitCode=60
Nov 02 15:30:18.304468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.304420    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd" event={"ID":"9fba93da-7b0c-4456-8c69-461cfbb3fca9","Type":"ContainerDied","Data":"b18e0d3441c37051afe5c63f6f91181224b1af020b0b2572a4f4132f2b81440b"}
Nov 02 15:30:18.304798 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.304777    4891 scope.go:122] "RemoveContainer" containerID="b18e0d3441c37051afe5c63f6f91181224b1af020b0b2572a4f4132f2b81440b"
Nov 02 15:30:18.305148 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:18.305123    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd_odh-ai-gateway-infra(9fba93da-7b0c-4456-8c69-461cfbb3fca9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9"
Nov 02 15:30:18.306590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.306570    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9" event={"ID":"96c01d26-e151-44a6-8501-d9618ad61ab9","Type":"ContainerStarted","Data":"d329f5e7562e134221de56fd2c6f0264abb7cf4803c13a70954f2880595f83db"}
Nov 02 15:30:18.313413 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.313383    4891 scope.go:122] "RemoveContainer" containerID="b12e91f483de481f3953caf8b014e964d7401bad502e795a8c443ba91e5be86f"
Nov 02 15:30:18.516304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.515584    4891 scope.go:122] "RemoveContainer" containerID="562886e777c691808b9124c60457aed6b63c54b1c8a0dbf5768e5d92f9d84bc7"
Nov 02 15:30:18.516304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.515822    4891 scope.go:122] "RemoveContainer" containerID="0d03dab745986c4edde96aa071c82d20f15cbe6221e9f5d58cd29c717b2e49d7"
Nov 02 15:30:18.521447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.521329    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"
Nov 02 15:30:18.523430 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.523413    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"
Nov 02 15:30:18.527773 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.527752    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82215bab-dd0f-4a3c-b9db-e42329343e34" path="/var/lib/kubelet/pods/82215bab-dd0f-4a3c-b9db-e42329343e34/volumes"
Nov 02 15:30:18.611980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.611944    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e212aa5a-ae15-4ac4-8958-d38c83b2658f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e212aa5a-ae15-4ac4-8958-d38c83b2658f-maas-api-service-ca\") pod \"e212aa5a-ae15-4ac4-8958-d38c83b2658f\" (UID: \"e212aa5a-ae15-4ac4-8958-d38c83b2658f\") "
Nov 02 15:30:18.612118 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.612055    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33e9cd6-6338-4897-9adc-8282a614e231-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33e9cd6-6338-4897-9adc-8282a614e231-maas-api-service-ca\") pod \"f33e9cd6-6338-4897-9adc-8282a614e231\" (UID: \"f33e9cd6-6338-4897-9adc-8282a614e231\") "
Nov 02 15:30:18.612118 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.612097    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e212aa5a-ae15-4ac4-8958-d38c83b2658f-kube-api-access-26pvv\" (UniqueName: \"kubernetes.io/projected/e212aa5a-ae15-4ac4-8958-d38c83b2658f-kube-api-access-26pvv\") pod \"e212aa5a-ae15-4ac4-8958-d38c83b2658f\" (UID: \"e212aa5a-ae15-4ac4-8958-d38c83b2658f\") "
Nov 02 15:30:18.612217 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.612171    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33e9cd6-6338-4897-9adc-8282a614e231-kube-api-access-4d692\" (UniqueName: \"kubernetes.io/projected/f33e9cd6-6338-4897-9adc-8282a614e231-kube-api-access-4d692\") pod \"f33e9cd6-6338-4897-9adc-8282a614e231\" (UID: \"f33e9cd6-6338-4897-9adc-8282a614e231\") "
Nov 02 15:30:18.612416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.612386    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e212aa5a-ae15-4ac4-8958-d38c83b2658f-maas-api-service-ca" pod "e212aa5a-ae15-4ac4-8958-d38c83b2658f" (UID: "e212aa5a-ae15-4ac4-8958-d38c83b2658f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:18.612499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.612472    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33e9cd6-6338-4897-9adc-8282a614e231-maas-api-service-ca" pod "f33e9cd6-6338-4897-9adc-8282a614e231" (UID: "f33e9cd6-6338-4897-9adc-8282a614e231"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:18.613657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.613630    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e212aa5a-ae15-4ac4-8958-d38c83b2658f-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:18.613657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.613652    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33e9cd6-6338-4897-9adc-8282a614e231-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:18.614330 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.614291    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e212aa5a-ae15-4ac4-8958-d38c83b2658f-kube-api-access-26pvv" pod "e212aa5a-ae15-4ac4-8958-d38c83b2658f" (UID: "e212aa5a-ae15-4ac4-8958-d38c83b2658f"). InnerVolumeSpecName "kube-api-access-26pvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:18.614597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.614577    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33e9cd6-6338-4897-9adc-8282a614e231-kube-api-access-4d692" pod "f33e9cd6-6338-4897-9adc-8282a614e231" (UID: "f33e9cd6-6338-4897-9adc-8282a614e231"). InnerVolumeSpecName "kube-api-access-4d692". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:18.714882 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.714848    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26pvv\" (UniqueName: \"kubernetes.io/projected/e212aa5a-ae15-4ac4-8958-d38c83b2658f-kube-api-access-26pvv\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:18.714882 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.714880    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d692\" (UniqueName: \"kubernetes.io/projected/f33e9cd6-6338-4897-9adc-8282a614e231-kube-api-access-4d692\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:18.773097 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.773071    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"]
Nov 02 15:30:18.902282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:18.902239    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"]
Nov 02 15:30:19.317101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.317076    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"
Nov 02 15:30:19.317632 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.317108    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6" event={"ID":"f33e9cd6-6338-4897-9adc-8282a614e231","Type":"ContainerDied","Data":"af8ea3c1c889f3abfe967f5eb86dff327c4092414acaa60f37be66a1e146a417"}
Nov 02 15:30:19.317632 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.317169    4891 scope.go:122] "RemoveContainer" containerID="d0742a106d16097c0207fe041693a67c1e5f038e3b15f8ef0a07e1058caef759"
Nov 02 15:30:19.319440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.319418    4891 generic.go:356] "Generic (PLEG): container finished" podID="13461dec-71c7-4ffc-aabb-f84f820c4502" containerID="c519a79d813fdc8f35af2c47a3c6fc9a26d26307298fe2c7b6e3c9d3298a9201" exitCode=60
Nov 02 15:30:19.319561 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.319492    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6" event={"ID":"13461dec-71c7-4ffc-aabb-f84f820c4502","Type":"ContainerDied","Data":"c519a79d813fdc8f35af2c47a3c6fc9a26d26307298fe2c7b6e3c9d3298a9201"}
Nov 02 15:30:19.319986 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.319956    4891 scope.go:122] "RemoveContainer" containerID="c519a79d813fdc8f35af2c47a3c6fc9a26d26307298fe2c7b6e3c9d3298a9201"
Nov 02 15:30:19.320379 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:19.320342    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6_odh-ai-gateway-infra(13461dec-71c7-4ffc-aabb-f84f820c4502)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502"
Nov 02 15:30:19.327338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.326631    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp" event={"ID":"d59415e5-4268-42af-b993-3eea7e047ef5","Type":"ContainerStarted","Data":"9f9eeff5822a19caa313622cdbba8fb605edc12a97d7a8efbaff9dfebb7354e0"}
Nov 02 15:30:19.330938 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.330911    4891 generic.go:356] "Generic (PLEG): container finished" podID="96c01d26-e151-44a6-8501-d9618ad61ab9" containerID="d329f5e7562e134221de56fd2c6f0264abb7cf4803c13a70954f2880595f83db" exitCode=60
Nov 02 15:30:19.331059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.331042    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9" event={"ID":"96c01d26-e151-44a6-8501-d9618ad61ab9","Type":"ContainerDied","Data":"d329f5e7562e134221de56fd2c6f0264abb7cf4803c13a70954f2880595f83db"}
Nov 02 15:30:19.331515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.331496    4891 scope.go:122] "RemoveContainer" containerID="d329f5e7562e134221de56fd2c6f0264abb7cf4803c13a70954f2880595f83db"
Nov 02 15:30:19.331931 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:19.331907    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9_odh-ai-gateway-infra(96c01d26-e151-44a6-8501-d9618ad61ab9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9"
Nov 02 15:30:19.334864 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.334838    4891 generic.go:356] "Generic (PLEG): container finished" podID="84777af8-4c53-4f7e-9a1a-940e6798353e" containerID="52a4aaa2ffefae8c5e38c63cd787447d5736ca35cc394750ff93a3d4a69c97b9" exitCode=60
Nov 02 15:30:19.334979 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.334965    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb" event={"ID":"84777af8-4c53-4f7e-9a1a-940e6798353e","Type":"ContainerDied","Data":"52a4aaa2ffefae8c5e38c63cd787447d5736ca35cc394750ff93a3d4a69c97b9"}
Nov 02 15:30:19.335414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.335392    4891 scope.go:122] "RemoveContainer" containerID="52a4aaa2ffefae8c5e38c63cd787447d5736ca35cc394750ff93a3d4a69c97b9"
Nov 02 15:30:19.335890 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.335657    4891 scope.go:122] "RemoveContainer" containerID="922587226e4ed06b06183367b66859dc484d6ff77a151c146333c0cb26faf40c"
Nov 02 15:30:19.335890 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:19.335798    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb_odh-ai-gateway-infra(84777af8-4c53-4f7e-9a1a-940e6798353e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e"
Nov 02 15:30:19.337813 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.336984    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb" event={"ID":"e212aa5a-ae15-4ac4-8958-d38c83b2658f","Type":"ContainerDied","Data":"5935660c591ecfb7fcc2b3987ff8577febcd00558314bb2feaad14ccba3e0999"}
Nov 02 15:30:19.337813 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.337386    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"
Nov 02 15:30:19.339941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.339922    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg" event={"ID":"9a174897-47d1-4a4c-994a-7186e7e405ef","Type":"ContainerStarted","Data":"027e279510f6e8a67cd1a383837d1561e8fc959a799e56245dd7d577285a0803"}
Nov 02 15:30:19.342240 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.342217    4891 generic.go:356] "Generic (PLEG): container finished" podID="585afb52-1760-4ed3-8fbb-e0e5e3077372" containerID="db03a4f7b4991179b4df0fdd9d2804f9ed048459b65e4c614ea92d2fcbc36d17" exitCode=60
Nov 02 15:30:19.342453 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.342287    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7" event={"ID":"585afb52-1760-4ed3-8fbb-e0e5e3077372","Type":"ContainerDied","Data":"db03a4f7b4991179b4df0fdd9d2804f9ed048459b65e4c614ea92d2fcbc36d17"}
Nov 02 15:30:19.342688 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.342670    4891 scope.go:122] "RemoveContainer" containerID="db03a4f7b4991179b4df0fdd9d2804f9ed048459b65e4c614ea92d2fcbc36d17"
Nov 02 15:30:19.342942 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:19.342920    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7_odh-ai-gateway-infra(585afb52-1760-4ed3-8fbb-e0e5e3077372)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372"
Nov 02 15:30:19.353656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.353606    4891 scope.go:122] "RemoveContainer" containerID="e9d6b034923f1f36e415635b7a3537ec78690a6067a778903f319acd8129c404"
Nov 02 15:30:19.375857 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.375829    4891 scope.go:122] "RemoveContainer" containerID="31cb1eea77d9bc4583b39d7e316d2f774f53783fa645d1689564ca111ba63a00"
Nov 02 15:30:19.431597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.431563    4891 scope.go:122] "RemoveContainer" containerID="39b92fc361decb16d04335247c81dc5069c8e32c15607f94ffa658ef4d4f205b"
Nov 02 15:30:19.440936 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.440905    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"]
Nov 02 15:30:19.451093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.451071    4891 scope.go:122] "RemoveContainer" containerID="ce1d8dd68328efbf31370b799c94604bf2d348dfdd69791158de61a29a2e00c7"
Nov 02 15:30:19.492974 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.492945    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"
Nov 02 15:30:19.518346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.514896    4891 scope.go:122] "RemoveContainer" containerID="6813c8ae65c09edb8dcb52cdfa47de281762fc029cc3ab0d7be4f38998db157c"
Nov 02 15:30:19.518346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.514941    4891 scope.go:122] "RemoveContainer" containerID="c7c4aa825a60297e73d1c46ae6975268a1c2d4b1561cecedc5d8c91588544907"
Nov 02 15:30:19.535408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.535374    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fba93da-7b0c-4456-8c69-461cfbb3fca9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fba93da-7b0c-4456-8c69-461cfbb3fca9-maas-api-service-ca\") pod \"9fba93da-7b0c-4456-8c69-461cfbb3fca9\" (UID: \"9fba93da-7b0c-4456-8c69-461cfbb3fca9\") "
Nov 02 15:30:19.535566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.535423    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fba93da-7b0c-4456-8c69-461cfbb3fca9-kube-api-access-gjdw2\" (UniqueName: \"kubernetes.io/projected/9fba93da-7b0c-4456-8c69-461cfbb3fca9-kube-api-access-gjdw2\") pod \"9fba93da-7b0c-4456-8c69-461cfbb3fca9\" (UID: \"9fba93da-7b0c-4456-8c69-461cfbb3fca9\") "
Nov 02 15:30:19.535823 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.535792    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fba93da-7b0c-4456-8c69-461cfbb3fca9-maas-api-service-ca" pod "9fba93da-7b0c-4456-8c69-461cfbb3fca9" (UID: "9fba93da-7b0c-4456-8c69-461cfbb3fca9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:19.536201 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.536175    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fba93da-7b0c-4456-8c69-461cfbb3fca9-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:19.540658 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.540627    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fba93da-7b0c-4456-8c69-461cfbb3fca9-kube-api-access-gjdw2" pod "9fba93da-7b0c-4456-8c69-461cfbb3fca9" (UID: "9fba93da-7b0c-4456-8c69-461cfbb3fca9"). InnerVolumeSpecName "kube-api-access-gjdw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:19.623127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.623102    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"
Nov 02 15:30:19.637527 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.637494    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdw2\" (UniqueName: \"kubernetes.io/projected/9fba93da-7b0c-4456-8c69-461cfbb3fca9-kube-api-access-gjdw2\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:19.662774 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.662745    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"]
Nov 02 15:30:19.674295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.674268    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7" podUID="8ecc4598-19ce-463b-a3de-5ee6ba41a08f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:30:19.674446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.674298    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r" podUID="7390b194-bcc2-4020-98c4-73cfb8d07480" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:30:19.738747 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.738713    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae86072-ec09-40f3-93f1-452f118684ba-kube-api-access-m5wwm\" (UniqueName: \"kubernetes.io/projected/2ae86072-ec09-40f3-93f1-452f118684ba-kube-api-access-m5wwm\") pod \"2ae86072-ec09-40f3-93f1-452f118684ba\" (UID: \"2ae86072-ec09-40f3-93f1-452f118684ba\") "
Nov 02 15:30:19.738919 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.738899    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae86072-ec09-40f3-93f1-452f118684ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae86072-ec09-40f3-93f1-452f118684ba-maas-api-service-ca\") pod \"2ae86072-ec09-40f3-93f1-452f118684ba\" (UID: \"2ae86072-ec09-40f3-93f1-452f118684ba\") "
Nov 02 15:30:19.739270 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.739247    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae86072-ec09-40f3-93f1-452f118684ba-maas-api-service-ca" pod "2ae86072-ec09-40f3-93f1-452f118684ba" (UID: "2ae86072-ec09-40f3-93f1-452f118684ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:19.745064 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.744791    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae86072-ec09-40f3-93f1-452f118684ba-kube-api-access-m5wwm" pod "2ae86072-ec09-40f3-93f1-452f118684ba" (UID: "2ae86072-ec09-40f3-93f1-452f118684ba"). InnerVolumeSpecName "kube-api-access-m5wwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:19.840623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.840595    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5wwm\" (UniqueName: \"kubernetes.io/projected/2ae86072-ec09-40f3-93f1-452f118684ba-kube-api-access-m5wwm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:19.840623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.840622    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae86072-ec09-40f3-93f1-452f118684ba-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:19.899559 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:19.899523    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"]
Nov 02 15:30:20.020661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.020626    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"]
Nov 02 15:30:20.282263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.282229    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"]
Nov 02 15:30:20.286561 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.286532    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de35cd-29893890-f89cb"]
Nov 02 15:30:20.358432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.358404    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9" event={"ID":"db707186-6796-4dad-893d-cb8ced999821","Type":"ContainerStarted","Data":"c09708440e6d06e3a3e1659ee215a825dd9737b1c7a0076bf72471e2e720493b"}
Nov 02 15:30:20.360364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.360337    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls" event={"ID":"de5a0c88-2c43-48b1-b966-edb2603ef847","Type":"ContainerStarted","Data":"9dfc3e20217da822a766f1eeea534b08a5566aa1e93bb495d47fc664725c7e04"}
Nov 02 15:30:20.362412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.362387    4891 generic.go:356] "Generic (PLEG): container finished" podID="9a174897-47d1-4a4c-994a-7186e7e405ef" containerID="027e279510f6e8a67cd1a383837d1561e8fc959a799e56245dd7d577285a0803" exitCode=60
Nov 02 15:30:20.362532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.362463    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg" event={"ID":"9a174897-47d1-4a4c-994a-7186e7e405ef","Type":"ContainerDied","Data":"027e279510f6e8a67cd1a383837d1561e8fc959a799e56245dd7d577285a0803"}
Nov 02 15:30:20.362592 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.362530    4891 scope.go:122] "RemoveContainer" containerID="0d03dab745986c4edde96aa071c82d20f15cbe6221e9f5d58cd29c717b2e49d7"
Nov 02 15:30:20.362881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.362867    4891 scope.go:122] "RemoveContainer" containerID="027e279510f6e8a67cd1a383837d1561e8fc959a799e56245dd7d577285a0803"
Nov 02 15:30:20.363131 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:20.363114    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg_odh-ai-gateway-infra(9a174897-47d1-4a4c-994a-7186e7e405ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef"
Nov 02 15:30:20.367521 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.367497    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"
Nov 02 15:30:20.367521 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.367514    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb" event={"ID":"2ae86072-ec09-40f3-93f1-452f118684ba","Type":"ContainerDied","Data":"1439e23ae5733c8f76d0f96f0c36f13428e1c75bf21cb2d0148aca995e39dfe0"}
Nov 02 15:30:20.373509 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.373476    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd" event={"ID":"9fba93da-7b0c-4456-8c69-461cfbb3fca9","Type":"ContainerDied","Data":"ec6fde9f396923663662372e785dcd55b90643d3881e9313419934b527d7c7bb"}
Nov 02 15:30:20.373622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.373594    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"
Nov 02 15:30:20.376751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.376716    4891 generic.go:356] "Generic (PLEG): container finished" podID="d59415e5-4268-42af-b993-3eea7e047ef5" containerID="9f9eeff5822a19caa313622cdbba8fb605edc12a97d7a8efbaff9dfebb7354e0" exitCode=60
Nov 02 15:30:20.377001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.376811    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp" event={"ID":"d59415e5-4268-42af-b993-3eea7e047ef5","Type":"ContainerDied","Data":"9f9eeff5822a19caa313622cdbba8fb605edc12a97d7a8efbaff9dfebb7354e0"}
Nov 02 15:30:20.377393 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.377348    4891 scope.go:122] "RemoveContainer" containerID="9f9eeff5822a19caa313622cdbba8fb605edc12a97d7a8efbaff9dfebb7354e0"
Nov 02 15:30:20.377710 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:20.377663    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp_odh-ai-gateway-infra(d59415e5-4268-42af-b993-3eea7e047ef5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp" podUID="d59415e5-4268-42af-b993-3eea7e047ef5"
Nov 02 15:30:20.383855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.383482    4891 scope.go:122] "RemoveContainer" containerID="92ae09ce6f48795fd3008f66e7b819b4b5a0556716b88a386ce14dd4ac56184f"
Nov 02 15:30:20.416228 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.415146    4891 scope.go:122] "RemoveContainer" containerID="b18e0d3441c37051afe5c63f6f91181224b1af020b0b2572a4f4132f2b81440b"
Nov 02 15:30:20.431531 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.431479    4891 scope.go:122] "RemoveContainer" containerID="562886e777c691808b9124c60457aed6b63c54b1c8a0dbf5768e5d92f9d84bc7"
Nov 02 15:30:20.542844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.542195    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" path="/var/lib/kubelet/pods/e212aa5a-ae15-4ac4-8958-d38c83b2658f/volumes"
Nov 02 15:30:20.637285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.637260    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"
Nov 02 15:30:20.641936 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.641905    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"
Nov 02 15:30:20.675182 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.675149    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"]
Nov 02 15:30:20.680281 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.680253    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fd1339-29893890-wnxp6"]
Nov 02 15:30:20.696326 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.696292    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"
Nov 02 15:30:20.727492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.727460    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"
Nov 02 15:30:20.754281 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.754249    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585afb52-1760-4ed3-8fbb-e0e5e3077372-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/585afb52-1760-4ed3-8fbb-e0e5e3077372-maas-api-service-ca\") pod \"585afb52-1760-4ed3-8fbb-e0e5e3077372\" (UID: \"585afb52-1760-4ed3-8fbb-e0e5e3077372\") "
Nov 02 15:30:20.754494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.754362    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84777af8-4c53-4f7e-9a1a-940e6798353e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84777af8-4c53-4f7e-9a1a-940e6798353e-maas-api-service-ca\") pod \"84777af8-4c53-4f7e-9a1a-940e6798353e\" (UID: \"84777af8-4c53-4f7e-9a1a-940e6798353e\") "
Nov 02 15:30:20.754494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.754384    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84777af8-4c53-4f7e-9a1a-940e6798353e-kube-api-access-ddzz2\" (UniqueName: \"kubernetes.io/projected/84777af8-4c53-4f7e-9a1a-940e6798353e-kube-api-access-ddzz2\") pod \"84777af8-4c53-4f7e-9a1a-940e6798353e\" (UID: \"84777af8-4c53-4f7e-9a1a-940e6798353e\") "
Nov 02 15:30:20.754620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.754555    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585afb52-1760-4ed3-8fbb-e0e5e3077372-kube-api-access-659jr\" (UniqueName: \"kubernetes.io/projected/585afb52-1760-4ed3-8fbb-e0e5e3077372-kube-api-access-659jr\") pod \"585afb52-1760-4ed3-8fbb-e0e5e3077372\" (UID: \"585afb52-1760-4ed3-8fbb-e0e5e3077372\") "
Nov 02 15:30:20.754620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.754600    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13461dec-71c7-4ffc-aabb-f84f820c4502-kube-api-access-bhdrf\" (UniqueName: \"kubernetes.io/projected/13461dec-71c7-4ffc-aabb-f84f820c4502-kube-api-access-bhdrf\") pod \"13461dec-71c7-4ffc-aabb-f84f820c4502\" (UID: \"13461dec-71c7-4ffc-aabb-f84f820c4502\") "
Nov 02 15:30:20.754730 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.754632    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13461dec-71c7-4ffc-aabb-f84f820c4502-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13461dec-71c7-4ffc-aabb-f84f820c4502-maas-api-service-ca\") pod \"13461dec-71c7-4ffc-aabb-f84f820c4502\" (UID: \"13461dec-71c7-4ffc-aabb-f84f820c4502\") "
Nov 02 15:30:20.754730 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.754710    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84777af8-4c53-4f7e-9a1a-940e6798353e-maas-api-service-ca" pod "84777af8-4c53-4f7e-9a1a-940e6798353e" (UID: "84777af8-4c53-4f7e-9a1a-940e6798353e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:20.754838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.754707    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585afb52-1760-4ed3-8fbb-e0e5e3077372-maas-api-service-ca" pod "585afb52-1760-4ed3-8fbb-e0e5e3077372" (UID: "585afb52-1760-4ed3-8fbb-e0e5e3077372"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:20.755067 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.755041    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13461dec-71c7-4ffc-aabb-f84f820c4502-maas-api-service-ca" pod "13461dec-71c7-4ffc-aabb-f84f820c4502" (UID: "13461dec-71c7-4ffc-aabb-f84f820c4502"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:20.755225 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.755199    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13461dec-71c7-4ffc-aabb-f84f820c4502-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:20.755338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.755227    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/585afb52-1760-4ed3-8fbb-e0e5e3077372-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:20.755338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.755241    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84777af8-4c53-4f7e-9a1a-940e6798353e-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:20.756620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.756599    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84777af8-4c53-4f7e-9a1a-940e6798353e-kube-api-access-ddzz2" pod "84777af8-4c53-4f7e-9a1a-940e6798353e" (UID: "84777af8-4c53-4f7e-9a1a-940e6798353e"). InnerVolumeSpecName "kube-api-access-ddzz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:20.756700 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.756687    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585afb52-1760-4ed3-8fbb-e0e5e3077372-kube-api-access-659jr" pod "585afb52-1760-4ed3-8fbb-e0e5e3077372" (UID: "585afb52-1760-4ed3-8fbb-e0e5e3077372"). InnerVolumeSpecName "kube-api-access-659jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:20.756777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.756758    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13461dec-71c7-4ffc-aabb-f84f820c4502-kube-api-access-bhdrf" pod "13461dec-71c7-4ffc-aabb-f84f820c4502" (UID: "13461dec-71c7-4ffc-aabb-f84f820c4502"). InnerVolumeSpecName "kube-api-access-bhdrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:20.856126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.856036    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c01d26-e151-44a6-8501-d9618ad61ab9-kube-api-access-mshbf\" (UniqueName: \"kubernetes.io/projected/96c01d26-e151-44a6-8501-d9618ad61ab9-kube-api-access-mshbf\") pod \"96c01d26-e151-44a6-8501-d9618ad61ab9\" (UID: \"96c01d26-e151-44a6-8501-d9618ad61ab9\") "
Nov 02 15:30:20.856126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.856118    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c01d26-e151-44a6-8501-d9618ad61ab9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c01d26-e151-44a6-8501-d9618ad61ab9-maas-api-service-ca\") pod \"96c01d26-e151-44a6-8501-d9618ad61ab9\" (UID: \"96c01d26-e151-44a6-8501-d9618ad61ab9\") "
Nov 02 15:30:20.856479 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.856463    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddzz2\" (UniqueName: \"kubernetes.io/projected/84777af8-4c53-4f7e-9a1a-940e6798353e-kube-api-access-ddzz2\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:20.856479 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.856479    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-659jr\" (UniqueName: \"kubernetes.io/projected/585afb52-1760-4ed3-8fbb-e0e5e3077372-kube-api-access-659jr\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:20.856602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.856490    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhdrf\" (UniqueName: \"kubernetes.io/projected/13461dec-71c7-4ffc-aabb-f84f820c4502-kube-api-access-bhdrf\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:20.856602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.856557    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c01d26-e151-44a6-8501-d9618ad61ab9-maas-api-service-ca" pod "96c01d26-e151-44a6-8501-d9618ad61ab9" (UID: "96c01d26-e151-44a6-8501-d9618ad61ab9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:20.858201 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.858180    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c01d26-e151-44a6-8501-d9618ad61ab9-kube-api-access-mshbf" pod "96c01d26-e151-44a6-8501-d9618ad61ab9" (UID: "96c01d26-e151-44a6-8501-d9618ad61ab9"). InnerVolumeSpecName "kube-api-access-mshbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:20.957409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.957378    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mshbf\" (UniqueName: \"kubernetes.io/projected/96c01d26-e151-44a6-8501-d9618ad61ab9-kube-api-access-mshbf\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:20.957409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:20.957403    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c01d26-e151-44a6-8501-d9618ad61ab9-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:21.181375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.181260    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"]
Nov 02 15:30:21.396792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.396758    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6" event={"ID":"13461dec-71c7-4ffc-aabb-f84f820c4502","Type":"ContainerDied","Data":"ca59401fa87c0ee4249109a7cef8f3f43fe6b7ee56ff2d2c2a9ec7d191d22c24"}
Nov 02 15:30:21.397347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.396829    4891 scope.go:122] "RemoveContainer" containerID="c519a79d813fdc8f35af2c47a3c6fc9a26d26307298fe2c7b6e3c9d3298a9201"
Nov 02 15:30:21.397347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.396764    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"
Nov 02 15:30:21.401082 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.401048    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9" event={"ID":"96c01d26-e151-44a6-8501-d9618ad61ab9","Type":"ContainerDied","Data":"7e61229be1b17ebcc8e141a52aab20f20e1a86726fb89f7b0b3d77184a597cf4"}
Nov 02 15:30:21.401234 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.401063    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"
Nov 02 15:30:21.402734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.402705    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb" event={"ID":"84777af8-4c53-4f7e-9a1a-940e6798353e","Type":"ContainerDied","Data":"5bd2418f4b78fa0dd516ef33324174346f77d5ac13801b8db1b6ca06720e0de8"}
Nov 02 15:30:21.402858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.402737    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"
Nov 02 15:30:21.404805 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.404781    4891 generic.go:356] "Generic (PLEG): container finished" podID="db707186-6796-4dad-893d-cb8ced999821" containerID="c09708440e6d06e3a3e1659ee215a825dd9737b1c7a0076bf72471e2e720493b" exitCode=60
Nov 02 15:30:21.404935 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.404855    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9" event={"ID":"db707186-6796-4dad-893d-cb8ced999821","Type":"ContainerDied","Data":"c09708440e6d06e3a3e1659ee215a825dd9737b1c7a0076bf72471e2e720493b"}
Nov 02 15:30:21.405295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.405271    4891 scope.go:122] "RemoveContainer" containerID="c09708440e6d06e3a3e1659ee215a825dd9737b1c7a0076bf72471e2e720493b"
Nov 02 15:30:21.405639 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:21.405614    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9_odh-ai-gateway-infra(db707186-6796-4dad-893d-cb8ced999821)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9" podUID="db707186-6796-4dad-893d-cb8ced999821"
Nov 02 15:30:21.407141 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.407122    4891 generic.go:356] "Generic (PLEG): container finished" podID="de5a0c88-2c43-48b1-b966-edb2603ef847" containerID="9dfc3e20217da822a766f1eeea534b08a5566aa1e93bb495d47fc664725c7e04" exitCode=60
Nov 02 15:30:21.407232 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.407193    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls" event={"ID":"de5a0c88-2c43-48b1-b966-edb2603ef847","Type":"ContainerDied","Data":"9dfc3e20217da822a766f1eeea534b08a5566aa1e93bb495d47fc664725c7e04"}
Nov 02 15:30:21.407937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.407624    4891 scope.go:122] "RemoveContainer" containerID="9dfc3e20217da822a766f1eeea534b08a5566aa1e93bb495d47fc664725c7e04"
Nov 02 15:30:21.407937 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:21.407913    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls_odh-ai-gateway-infra(de5a0c88-2c43-48b1-b966-edb2603ef847)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847"
Nov 02 15:30:21.410496 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.410478    4891 scope.go:122] "RemoveContainer" containerID="d329f5e7562e134221de56fd2c6f0264abb7cf4803c13a70954f2880595f83db"
Nov 02 15:30:21.416215 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.410904    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7" event={"ID":"585afb52-1760-4ed3-8fbb-e0e5e3077372","Type":"ContainerDied","Data":"536e05acb6fd9736f694805077809fc1174ec76f759ce6b5d39b12e03f0c2371"}
Nov 02 15:30:21.416215 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.410921    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"
Nov 02 15:30:21.419267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.419237    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"]
Nov 02 15:30:21.428183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.428163    4891 scope.go:122] "RemoveContainer" containerID="52a4aaa2ffefae8c5e38c63cd787447d5736ca35cc394750ff93a3d4a69c97b9"
Nov 02 15:30:21.440980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.440892    4891 scope.go:122] "RemoveContainer" containerID="c7c4aa825a60297e73d1c46ae6975268a1c2d4b1561cecedc5d8c91588544907"
Nov 02 15:30:21.484669 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.484507    4891 scope.go:122] "RemoveContainer" containerID="6813c8ae65c09edb8dcb52cdfa47de281762fc029cc3ab0d7be4f38998db157c"
Nov 02 15:30:21.502337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.502084    4891 scope.go:122] "RemoveContainer" containerID="db03a4f7b4991179b4df0fdd9d2804f9ed048459b65e4c614ea92d2fcbc36d17"
Nov 02 15:30:21.571425 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.571399    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"
Nov 02 15:30:21.649852 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.649827    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"
Nov 02 15:30:21.659941 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.659914    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"]
Nov 02 15:30:21.666350 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.666328    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a174897-47d1-4a4c-994a-7186e7e405ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a174897-47d1-4a4c-994a-7186e7e405ef-maas-api-service-ca\") pod \"9a174897-47d1-4a4c-994a-7186e7e405ef\" (UID: \"9a174897-47d1-4a4c-994a-7186e7e405ef\") "
Nov 02 15:30:21.666425 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.666375    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a174897-47d1-4a4c-994a-7186e7e405ef-kube-api-access-8qcs8\" (UniqueName: \"kubernetes.io/projected/9a174897-47d1-4a4c-994a-7186e7e405ef-kube-api-access-8qcs8\") pod \"9a174897-47d1-4a4c-994a-7186e7e405ef\" (UID: \"9a174897-47d1-4a4c-994a-7186e7e405ef\") "
Nov 02 15:30:21.666649 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.666629    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a174897-47d1-4a4c-994a-7186e7e405ef-maas-api-service-ca" pod "9a174897-47d1-4a4c-994a-7186e7e405ef" (UID: "9a174897-47d1-4a4c-994a-7186e7e405ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:21.666903 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.666888    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a174897-47d1-4a4c-994a-7186e7e405ef-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:21.668414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.668396    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a174897-47d1-4a4c-994a-7186e7e405ef-kube-api-access-8qcs8" pod "9a174897-47d1-4a4c-994a-7186e7e405ef" (UID: "9a174897-47d1-4a4c-994a-7186e7e405ef"). InnerVolumeSpecName "kube-api-access-8qcs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:21.675982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.675952    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6dc69b-29893890-m7vnb"]
Nov 02 15:30:21.767871 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.767788    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59415e5-4268-42af-b993-3eea7e047ef5-kube-api-access-49j95\" (UniqueName: \"kubernetes.io/projected/d59415e5-4268-42af-b993-3eea7e047ef5-kube-api-access-49j95\") pod \"d59415e5-4268-42af-b993-3eea7e047ef5\" (UID: \"d59415e5-4268-42af-b993-3eea7e047ef5\") "
Nov 02 15:30:21.767871 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.767837    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59415e5-4268-42af-b993-3eea7e047ef5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59415e5-4268-42af-b993-3eea7e047ef5-maas-api-service-ca\") pod \"d59415e5-4268-42af-b993-3eea7e047ef5\" (UID: \"d59415e5-4268-42af-b993-3eea7e047ef5\") "
Nov 02 15:30:21.768139 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.768110    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qcs8\" (UniqueName: \"kubernetes.io/projected/9a174897-47d1-4a4c-994a-7186e7e405ef-kube-api-access-8qcs8\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:21.768269 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.768245    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59415e5-4268-42af-b993-3eea7e047ef5-maas-api-service-ca" pod "d59415e5-4268-42af-b993-3eea7e047ef5" (UID: "d59415e5-4268-42af-b993-3eea7e047ef5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:21.769894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.769874    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59415e5-4268-42af-b993-3eea7e047ef5-kube-api-access-49j95" pod "d59415e5-4268-42af-b993-3eea7e047ef5" (UID: "d59415e5-4268-42af-b993-3eea7e047ef5"). InnerVolumeSpecName "kube-api-access-49j95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:21.849549 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.849510    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"]
Nov 02 15:30:21.853622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.853589    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-468af4-29893890-m5xfd"]
Nov 02 15:30:21.869068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.869042    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49j95\" (UniqueName: \"kubernetes.io/projected/d59415e5-4268-42af-b993-3eea7e047ef5-kube-api-access-49j95\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:21.869068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:21.869066    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59415e5-4268-42af-b993-3eea7e047ef5-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:22.026158 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.026123    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"]
Nov 02 15:30:22.146043 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.146007    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"]
Nov 02 15:30:22.157081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.157045    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-895ac8-29893890-pslv9"]
Nov 02 15:30:22.328004 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.327918    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"]
Nov 02 15:30:22.334589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.334560    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-303ba8-29893890-6rvp6"]
Nov 02 15:30:22.371121 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.371088    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"]
Nov 02 15:30:22.435301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.435257    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp" event={"ID":"d59415e5-4268-42af-b993-3eea7e047ef5","Type":"ContainerDied","Data":"239f4db1e5f16df4dcdab3a90126e097bad9fd3db4dc06c6465dce73229adff8"}
Nov 02 15:30:22.435832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.435396    4891 scope.go:122] "RemoveContainer" containerID="9f9eeff5822a19caa313622cdbba8fb605edc12a97d7a8efbaff9dfebb7354e0"
Nov 02 15:30:22.436047 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.436029    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"
Nov 02 15:30:22.449169 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.449080    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg" event={"ID":"9a174897-47d1-4a4c-994a-7186e7e405ef","Type":"ContainerDied","Data":"d46d676c64499d40c26afb20eb8542699ad207cd8e31695aa40e8be389331a33"}
Nov 02 15:30:22.449258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.449215    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"
Nov 02 15:30:22.452301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.451538    4891 scope.go:122] "RemoveContainer" containerID="027e279510f6e8a67cd1a383837d1561e8fc959a799e56245dd7d577285a0803"
Nov 02 15:30:22.531583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.529758    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502" path="/var/lib/kubelet/pods/13461dec-71c7-4ffc-aabb-f84f820c4502/volumes"
Nov 02 15:30:22.531583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.530400    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae86072-ec09-40f3-93f1-452f118684ba" path="/var/lib/kubelet/pods/2ae86072-ec09-40f3-93f1-452f118684ba/volumes"
Nov 02 15:30:22.531583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.530925    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9" path="/var/lib/kubelet/pods/96c01d26-e151-44a6-8501-d9618ad61ab9/volumes"
Nov 02 15:30:22.531840 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.531622    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" path="/var/lib/kubelet/pods/9fba93da-7b0c-4456-8c69-461cfbb3fca9/volumes"
Nov 02 15:30:22.532072 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.532055    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33e9cd6-6338-4897-9adc-8282a614e231" path="/var/lib/kubelet/pods/f33e9cd6-6338-4897-9adc-8282a614e231/volumes"
Nov 02 15:30:22.571019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.570984    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"]
Nov 02 15:30:22.577884 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.577857    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b87e90-29893890-zd8wb"]
Nov 02 15:30:22.641446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.641414    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"
Nov 02 15:30:22.677612 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.677588    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"
Nov 02 15:30:22.791374 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.791339    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db707186-6796-4dad-893d-cb8ced999821-kube-api-access-t9hsq\" (UniqueName: \"kubernetes.io/projected/db707186-6796-4dad-893d-cb8ced999821-kube-api-access-t9hsq\") pod \"db707186-6796-4dad-893d-cb8ced999821\" (UID: \"db707186-6796-4dad-893d-cb8ced999821\") "
Nov 02 15:30:22.791571 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.791477    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5a0c88-2c43-48b1-b966-edb2603ef847-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5a0c88-2c43-48b1-b966-edb2603ef847-maas-api-service-ca\") pod \"de5a0c88-2c43-48b1-b966-edb2603ef847\" (UID: \"de5a0c88-2c43-48b1-b966-edb2603ef847\") "
Nov 02 15:30:22.791571 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.791569    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db707186-6796-4dad-893d-cb8ced999821-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db707186-6796-4dad-893d-cb8ced999821-maas-api-service-ca\") pod \"db707186-6796-4dad-893d-cb8ced999821\" (UID: \"db707186-6796-4dad-893d-cb8ced999821\") "
Nov 02 15:30:22.791715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.791612    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5a0c88-2c43-48b1-b966-edb2603ef847-kube-api-access-zwtxg\" (UniqueName: \"kubernetes.io/projected/de5a0c88-2c43-48b1-b966-edb2603ef847-kube-api-access-zwtxg\") pod \"de5a0c88-2c43-48b1-b966-edb2603ef847\" (UID: \"de5a0c88-2c43-48b1-b966-edb2603ef847\") "
Nov 02 15:30:22.791905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.791879    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5a0c88-2c43-48b1-b966-edb2603ef847-maas-api-service-ca" pod "de5a0c88-2c43-48b1-b966-edb2603ef847" (UID: "de5a0c88-2c43-48b1-b966-edb2603ef847"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:22.791991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.791945    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db707186-6796-4dad-893d-cb8ced999821-maas-api-service-ca" pod "db707186-6796-4dad-893d-cb8ced999821" (UID: "db707186-6796-4dad-893d-cb8ced999821"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:22.792048 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.792026    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5a0c88-2c43-48b1-b966-edb2603ef847-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:22.793496 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.793468    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db707186-6796-4dad-893d-cb8ced999821-kube-api-access-t9hsq" pod "db707186-6796-4dad-893d-cb8ced999821" (UID: "db707186-6796-4dad-893d-cb8ced999821"). InnerVolumeSpecName "kube-api-access-t9hsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:22.793778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.793754    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5a0c88-2c43-48b1-b966-edb2603ef847-kube-api-access-zwtxg" pod "de5a0c88-2c43-48b1-b966-edb2603ef847" (UID: "de5a0c88-2c43-48b1-b966-edb2603ef847"). InnerVolumeSpecName "kube-api-access-zwtxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:22.893256 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.893172    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwtxg\" (UniqueName: \"kubernetes.io/projected/de5a0c88-2c43-48b1-b966-edb2603ef847-kube-api-access-zwtxg\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:22.893256 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.893201    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9hsq\" (UniqueName: \"kubernetes.io/projected/db707186-6796-4dad-893d-cb8ced999821-kube-api-access-t9hsq\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:22.893256 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.893211    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db707186-6796-4dad-893d-cb8ced999821-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:22.922147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.922081    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"]
Nov 02 15:30:22.925781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:22.925756    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6b2f61-29893890-vg4f7"]
Nov 02 15:30:23.389426 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.389386    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"]
Nov 02 15:30:23.391368 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.391342    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-885c69-29893890-6fssg"]
Nov 02 15:30:23.478045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.478018    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"
Nov 02 15:30:23.478045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.478024    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9" event={"ID":"db707186-6796-4dad-893d-cb8ced999821","Type":"ContainerDied","Data":"b9d50b5f983af2caca29e4c1229b67acbd194e29cdc3816d7df4b383e28c055b"}
Nov 02 15:30:23.478599 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.478084    4891 scope.go:122] "RemoveContainer" containerID="c09708440e6d06e3a3e1659ee215a825dd9737b1c7a0076bf72471e2e720493b"
Nov 02 15:30:23.479708 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.479687    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"
Nov 02 15:30:23.479845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.479690    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls" event={"ID":"de5a0c88-2c43-48b1-b966-edb2603ef847","Type":"ContainerDied","Data":"3a2134eb475129efdf4355572ad38d80a2fa44ef88e7517d64641572585c995d"}
Nov 02 15:30:23.495672 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.495638    4891 scope.go:122] "RemoveContainer" containerID="9dfc3e20217da822a766f1eeea534b08a5566aa1e93bb495d47fc664725c7e04"
Nov 02 15:30:23.499469 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.499445    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"]
Nov 02 15:30:23.503983 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.503962    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db3931-29893890-585hp"]
Nov 02 15:30:23.609092 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.609062    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"]
Nov 02 15:30:23.614610 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.614582    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4c5b4e-29893890-cnsls"]
Nov 02 15:30:23.725594 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.725506    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"]
Nov 02 15:30:23.729123 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:23.729098    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-559973-29893890-wvzq9"]
Nov 02 15:30:24.524478 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:24.524443    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372" path="/var/lib/kubelet/pods/585afb52-1760-4ed3-8fbb-e0e5e3077372/volumes"
Nov 02 15:30:24.524954 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:24.524903    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e" path="/var/lib/kubelet/pods/84777af8-4c53-4f7e-9a1a-940e6798353e/volumes"
Nov 02 15:30:24.525354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:24.525339    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef" path="/var/lib/kubelet/pods/9a174897-47d1-4a4c-994a-7186e7e405ef/volumes"
Nov 02 15:30:24.526114 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:24.526098    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59415e5-4268-42af-b993-3eea7e047ef5" path="/var/lib/kubelet/pods/d59415e5-4268-42af-b993-3eea7e047ef5/volumes"
Nov 02 15:30:24.526519 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:24.526506    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db707186-6796-4dad-893d-cb8ced999821" path="/var/lib/kubelet/pods/db707186-6796-4dad-893d-cb8ced999821/volumes"
Nov 02 15:30:24.526880 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:24.526867    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847" path="/var/lib/kubelet/pods/de5a0c88-2c43-48b1-b966-edb2603ef847/volumes"
Nov 02 15:30:26.160709 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.160653    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"]
Nov 02 15:30:26.162983 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.162953    4891 cpu_manager.go:414] "containerMap: removing container" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e" containerName="cleanup"
Nov 02 15:30:26.162983 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.162981    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163005    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163013    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163029    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163036    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163048    4891 cpu_manager.go:414] "containerMap: removing container" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163056    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163071    4891 cpu_manager.go:414] "containerMap: removing container" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163080    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163089    4891 cpu_manager.go:414] "containerMap: removing container" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163096    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163111    4891 cpu_manager.go:414] "containerMap: removing container" podUID="db707186-6796-4dad-893d-cb8ced999821" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163117    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db707186-6796-4dad-893d-cb8ced999821" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163127    4891 cpu_manager.go:414] "containerMap: removing container" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163133    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163144    4891 cpu_manager.go:414] "containerMap: removing container" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163152    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163170    4891 cpu_manager.go:414] "containerMap: removing container" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163177    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163189    4891 cpu_manager.go:414] "containerMap: removing container" podUID="d59415e5-4268-42af-b993-3eea7e047ef5" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163196    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59415e5-4268-42af-b993-3eea7e047ef5" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163208    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" containerName="istio-proxy"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163215    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" containerName="istio-proxy"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163226    4891 cpu_manager.go:414] "containerMap: removing container" podUID="2ae86072-ec09-40f3-93f1-452f118684ba" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163232    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae86072-ec09-40f3-93f1-452f118684ba" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163240    4891 cpu_manager.go:414] "containerMap: removing container" podUID="d59415e5-4268-42af-b993-3eea7e047ef5" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163247    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59415e5-4268-42af-b993-3eea7e047ef5" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163257    4891 cpu_manager.go:414] "containerMap: removing container" podUID="db707186-6796-4dad-893d-cb8ced999821" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163264    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db707186-6796-4dad-893d-cb8ced999821" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163274    4891 cpu_manager.go:414] "containerMap: removing container" podUID="d59415e5-4268-42af-b993-3eea7e047ef5" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163280    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59415e5-4268-42af-b993-3eea7e047ef5" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163294    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163301    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163326    4891 cpu_manager.go:414] "containerMap: removing container" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163333    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163345    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f33e9cd6-6338-4897-9adc-8282a614e231" containerName="cleanup"
Nov 02 15:30:26.163291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163352    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33e9cd6-6338-4897-9adc-8282a614e231" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163373    4891 cpu_manager.go:414] "containerMap: removing container" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163380    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163390    4891 cpu_manager.go:414] "containerMap: removing container" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163396    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163406    4891 cpu_manager.go:414] "containerMap: removing container" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163413    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163429    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163435    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163447    4891 cpu_manager.go:414] "containerMap: removing container" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163453    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163465    4891 cpu_manager.go:414] "containerMap: removing container" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163472    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163484    4891 cpu_manager.go:414] "containerMap: removing container" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163490    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163500    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163506    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163516    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f33e9cd6-6338-4897-9adc-8282a614e231" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163523    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33e9cd6-6338-4897-9adc-8282a614e231" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163538    4891 cpu_manager.go:414] "containerMap: removing container" podUID="2ae86072-ec09-40f3-93f1-452f118684ba" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163545    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae86072-ec09-40f3-93f1-452f118684ba" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163561    4891 cpu_manager.go:414] "containerMap: removing container" podUID="7137ff72-f89d-46c8-95c7-34569d6cdd06" containerName="discovery"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163568    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7137ff72-f89d-46c8-95c7-34569d6cdd06" containerName="discovery"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163580    4891 cpu_manager.go:414] "containerMap: removing container" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163587    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163600    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163606    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163791    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163805    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae86072-ec09-40f3-93f1-452f118684ba" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163815    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163824    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163832    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163839    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163847    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163858    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163866    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163874    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ca96d6-a378-43b9-855f-29b40a217715" containerName="main"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163884    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163891    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33e9cd6-6338-4897-9adc-8282a614e231" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163899    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae86072-ec09-40f3-93f1-452f118684ba" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163908    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="db707186-6796-4dad-893d-cb8ced999821" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163917    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33e9cd6-6338-4897-9adc-8282a614e231" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163928    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163938    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="7137ff72-f89d-46c8-95c7-34569d6cdd06" containerName="discovery"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163946    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163955    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e14887d-6b64-456e-ad0b-3e0b34789cbe" containerName="main"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163962    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163975    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163983    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.163990    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d1d6d1-77ba-4dca-9b35-dbeb471cf2e5" containerName="istio-proxy"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.164000    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="db707186-6796-4dad-893d-cb8ced999821" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.164009    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.164018    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.164028    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59415e5-4268-42af-b993-3eea7e047ef5" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.164037    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bee6b2b-9e64-4b60-9f35-0775382a60f1" containerName="istio-proxy"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.164045    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.164053    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59415e5-4268-42af-b993-3eea7e047ef5" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.164061    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="db707186-6796-4dad-893d-cb8ced999821" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.164068    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59415e5-4268-42af-b993-3eea7e047ef5" containerName="cleanup"
Nov 02 15:30:26.165009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.164077    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" containerName="cleanup"
Nov 02 15:30:26.169566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.168409    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:26.221374 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.221343    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"]
Nov 02 15:30:26.240665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.240631    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-303ba8-747ns\" (UID: \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:26.240831 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.240784    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps4mj\" (UniqueName: \"kubernetes.io/projected/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-kube-api-access-ps4mj\") pod \"maas-api-revoke-keys-e2e-api-a-303ba8-747ns\" (UID: \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:26.241029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.241006    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-303ba8-747ns\" (UID: \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:26.266602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.266577    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:30:26.266768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.266634    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:30:26.266768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.266684    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:30:26.270385 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.270357    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:30:26.341876 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.341840    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-303ba8-747ns\" (UID: \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:26.342089 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.341930    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-303ba8-747ns\" (UID: \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:26.342089 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.341983    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps4mj\" (UniqueName: \"kubernetes.io/projected/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-kube-api-access-ps4mj\") pod \"maas-api-revoke-keys-e2e-api-a-303ba8-747ns\" (UID: \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:26.342684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.342661    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-303ba8-747ns\" (UID: \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:26.344274 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.344256    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-303ba8-747ns\" (UID: \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:26.403142 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.403106    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps4mj\" (UniqueName: \"kubernetes.io/projected/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-kube-api-access-ps4mj\") pod \"maas-api-revoke-keys-e2e-api-a-303ba8-747ns\" (UID: \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:26.489560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.489476    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:26.883290 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:26.883256    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"]
Nov 02 15:30:26.884352 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:30:26.884292    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7ee164_fd9d_4b6d_b4a1_9a30ff7095b5.slice/crio-7a1b68869141452dbb8d6752850e57fdee7658434dd18d7e191fc0f7a8b9aa61 WatchSource:0}: Error finding container 7a1b68869141452dbb8d6752850e57fdee7658434dd18d7e191fc0f7a8b9aa61: Status 404 returned error can't find the container with id 7a1b68869141452dbb8d6752850e57fdee7658434dd18d7e191fc0f7a8b9aa61
Nov 02 15:30:27.528612 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.528581    4891 generic.go:356] "Generic (PLEG): container finished" podID="8ecc4598-19ce-463b-a3de-5ee6ba41a08f" containerID="c0fb0e54f4541765d9fad7748a1e56e1f7ff63b36422f6124fd038189948f221" exitCode=0
Nov 02 15:30:27.528980 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.528658    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7" event={"ID":"8ecc4598-19ce-463b-a3de-5ee6ba41a08f","Type":"ContainerDied","Data":"c0fb0e54f4541765d9fad7748a1e56e1f7ff63b36422f6124fd038189948f221"}
Nov 02 15:30:27.530847 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.530744    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns" event={"ID":"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5","Type":"ContainerStarted","Data":"4b4493842e2c74ea36beefb3068ca2b3dd6a48191c0de8e86370e148c8c23bbc"}
Nov 02 15:30:27.530847 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.530803    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns" event={"ID":"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5","Type":"ContainerStarted","Data":"7a1b68869141452dbb8d6752850e57fdee7658434dd18d7e191fc0f7a8b9aa61"}
Nov 02 15:30:27.532815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.532793    4891 generic.go:356] "Generic (PLEG): container finished" podID="7390b194-bcc2-4020-98c4-73cfb8d07480" containerID="84b625e3d4ddb407046a35ddbecbb18be2ef00c46abe04b81b1b2fa67fd573f3" exitCode=0
Nov 02 15:30:27.532922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.532832    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r" event={"ID":"7390b194-bcc2-4020-98c4-73cfb8d07480","Type":"ContainerDied","Data":"84b625e3d4ddb407046a35ddbecbb18be2ef00c46abe04b81b1b2fa67fd573f3"}
Nov 02 15:30:27.532922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.532861    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r" event={"ID":"7390b194-bcc2-4020-98c4-73cfb8d07480","Type":"ContainerDied","Data":"0f2aea98457fb8113f6875ba2e1bd9471555540b5e4f5c837d8e985226c1c246"}
Nov 02 15:30:27.532922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.532876    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f2aea98457fb8113f6875ba2e1bd9471555540b5e4f5c837d8e985226c1c246"
Nov 02 15:30:27.549524 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.549502    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"
Nov 02 15:30:27.646623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.646598    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"
Nov 02 15:30:27.663417 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.663388    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7390b194-bcc2-4020-98c4-73cfb8d07480-kube-api-access-hkgnl\" (UniqueName: \"kubernetes.io/projected/7390b194-bcc2-4020-98c4-73cfb8d07480-kube-api-access-hkgnl\") pod \"7390b194-bcc2-4020-98c4-73cfb8d07480\" (UID: \"7390b194-bcc2-4020-98c4-73cfb8d07480\") "
Nov 02 15:30:27.663628 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.663611    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7390b194-bcc2-4020-98c4-73cfb8d07480-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7390b194-bcc2-4020-98c4-73cfb8d07480-plugins-config-volume\") pod \"7390b194-bcc2-4020-98c4-73cfb8d07480\" (UID: \"7390b194-bcc2-4020-98c4-73cfb8d07480\") "
Nov 02 15:30:27.664339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.663950    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7390b194-bcc2-4020-98c4-73cfb8d07480-plugins-config-volume" pod "7390b194-bcc2-4020-98c4-73cfb8d07480" (UID: "7390b194-bcc2-4020-98c4-73cfb8d07480"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:27.664339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.664284    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7390b194-bcc2-4020-98c4-73cfb8d07480-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:27.666180 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.666151    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7390b194-bcc2-4020-98c4-73cfb8d07480-kube-api-access-hkgnl" pod "7390b194-bcc2-4020-98c4-73cfb8d07480" (UID: "7390b194-bcc2-4020-98c4-73cfb8d07480"). InnerVolumeSpecName "kube-api-access-hkgnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:27.765035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.764953    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-plugins-config-volume\") pod \"8ecc4598-19ce-463b-a3de-5ee6ba41a08f\" (UID: \"8ecc4598-19ce-463b-a3de-5ee6ba41a08f\") "
Nov 02 15:30:27.765191 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.765075    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-kube-api-access-fcfhh\" (UniqueName: \"kubernetes.io/projected/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-kube-api-access-fcfhh\") pod \"8ecc4598-19ce-463b-a3de-5ee6ba41a08f\" (UID: \"8ecc4598-19ce-463b-a3de-5ee6ba41a08f\") "
Nov 02 15:30:27.765390 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.765366    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-plugins-config-volume" pod "8ecc4598-19ce-463b-a3de-5ee6ba41a08f" (UID: "8ecc4598-19ce-463b-a3de-5ee6ba41a08f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:27.765543 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.765521    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:27.765543 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.765541    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkgnl\" (UniqueName: \"kubernetes.io/projected/7390b194-bcc2-4020-98c4-73cfb8d07480-kube-api-access-hkgnl\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:27.767258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.767232    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-kube-api-access-fcfhh" pod "8ecc4598-19ce-463b-a3de-5ee6ba41a08f" (UID: "8ecc4598-19ce-463b-a3de-5ee6ba41a08f"). InnerVolumeSpecName "kube-api-access-fcfhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:27.804412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.804358    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns" podStartSLOduration=1.804341293 podStartE2EDuration="1.804341293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:27.685407265 +0000 UTC m=+2035.278024064" watchObservedRunningTime="2026-11-02 15:30:27.804341293 +0000 UTC m=+2035.396958094"
Nov 02 15:30:27.867197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:27.867167    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcfhh\" (UniqueName: \"kubernetes.io/projected/8ecc4598-19ce-463b-a3de-5ee6ba41a08f-kube-api-access-fcfhh\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:28.161784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.161746    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"]
Nov 02 15:30:28.162361 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162345    4891 cpu_manager.go:414] "containerMap: removing container" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372" containerName="cleanup"
Nov 02 15:30:28.162437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162362    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585afb52-1760-4ed3-8fbb-e0e5e3077372" containerName="cleanup"
Nov 02 15:30:28.162437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162378    4891 cpu_manager.go:414] "containerMap: removing container" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" containerName="cleanup"
Nov 02 15:30:28.162437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162385    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" containerName="cleanup"
Nov 02 15:30:28.162437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162397    4891 cpu_manager.go:414] "containerMap: removing container" podUID="2ae86072-ec09-40f3-93f1-452f118684ba" containerName="cleanup"
Nov 02 15:30:28.162437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162402    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae86072-ec09-40f3-93f1-452f118684ba" containerName="cleanup"
Nov 02 15:30:28.162437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162413    4891 cpu_manager.go:414] "containerMap: removing container" podUID="7390b194-bcc2-4020-98c4-73cfb8d07480" containerName="payload-processing"
Nov 02 15:30:28.162437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162418    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7390b194-bcc2-4020-98c4-73cfb8d07480" containerName="payload-processing"
Nov 02 15:30:28.162437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162431    4891 cpu_manager.go:414] "containerMap: removing container" podUID="8ecc4598-19ce-463b-a3de-5ee6ba41a08f" containerName="payload-pre-processing"
Nov 02 15:30:28.162437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162436    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ecc4598-19ce-463b-a3de-5ee6ba41a08f" containerName="payload-pre-processing"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162448    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162453    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162462    4891 cpu_manager.go:414] "containerMap: removing container" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162466    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162473    4891 cpu_manager.go:414] "containerMap: removing container" podUID="db707186-6796-4dad-893d-cb8ced999821" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162478    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db707186-6796-4dad-893d-cb8ced999821" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162485    4891 cpu_manager.go:414] "containerMap: removing container" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162489    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162499    4891 cpu_manager.go:414] "containerMap: removing container" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162504    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c01d26-e151-44a6-8501-d9618ad61ab9" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162515    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f33e9cd6-6338-4897-9adc-8282a614e231" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162520    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33e9cd6-6338-4897-9adc-8282a614e231" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162599    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fba93da-7b0c-4456-8c69-461cfbb3fca9" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162610    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="e212aa5a-ae15-4ac4-8958-d38c83b2658f" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162617    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="13461dec-71c7-4ffc-aabb-f84f820c4502" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162622    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a174897-47d1-4a4c-994a-7186e7e405ef" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162629    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae86072-ec09-40f3-93f1-452f118684ba" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162635    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33e9cd6-6338-4897-9adc-8282a614e231" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162642    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="7390b194-bcc2-4020-98c4-73cfb8d07480" containerName="payload-processing"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162648    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5a0c88-2c43-48b1-b966-edb2603ef847" containerName="cleanup"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162655    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ecc4598-19ce-463b-a3de-5ee6ba41a08f" containerName="payload-pre-processing"
Nov 02 15:30:28.162764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.162661    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="84777af8-4c53-4f7e-9a1a-940e6798353e" containerName="cleanup"
Nov 02 15:30:28.166005 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.165983    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:28.227257 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.227223    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"]
Nov 02 15:30:28.272170 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.272133    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq\" (UID: \"cf99298c-7016-4838-98eb-f729a6d2398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:28.272380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.272282    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq\" (UID: \"cf99298c-7016-4838-98eb-f729a6d2398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:28.272462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.272445    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjf7q\" (UniqueName: \"kubernetes.io/projected/cf99298c-7016-4838-98eb-f729a6d2398d-kube-api-access-vjf7q\") pod \"maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq\" (UID: \"cf99298c-7016-4838-98eb-f729a6d2398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:28.374050 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.374009    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq\" (UID: \"cf99298c-7016-4838-98eb-f729a6d2398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:28.374235 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.374140    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjf7q\" (UniqueName: \"kubernetes.io/projected/cf99298c-7016-4838-98eb-f729a6d2398d-kube-api-access-vjf7q\") pod \"maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq\" (UID: \"cf99298c-7016-4838-98eb-f729a6d2398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:28.374235 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.374197    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq\" (UID: \"cf99298c-7016-4838-98eb-f729a6d2398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:28.374846 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.374819    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq\" (UID: \"cf99298c-7016-4838-98eb-f729a6d2398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:28.376453 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.376429    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq\" (UID: \"cf99298c-7016-4838-98eb-f729a6d2398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:28.435454 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.435383    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjf7q\" (UniqueName: \"kubernetes.io/projected/cf99298c-7016-4838-98eb-f729a6d2398d-kube-api-access-vjf7q\") pod \"maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq\" (UID: \"cf99298c-7016-4838-98eb-f729a6d2398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:28.483489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.483459    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:28.543071 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.543044    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"
Nov 02 15:30:28.544525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.543046    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7" event={"ID":"8ecc4598-19ce-463b-a3de-5ee6ba41a08f","Type":"ContainerDied","Data":"2ff3d49a6a61557695260d785ffe9bd3d6cb00a4dbc2d1f8092b3f22ae9e0645"}
Nov 02 15:30:28.544525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.543242    4891 scope.go:122] "RemoveContainer" containerID="c0fb0e54f4541765d9fad7748a1e56e1f7ff63b36422f6124fd038189948f221"
Nov 02 15:30:28.548709 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.548485    4891 generic.go:356] "Generic (PLEG): container finished" podID="7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5" containerID="4b4493842e2c74ea36beefb3068ca2b3dd6a48191c0de8e86370e148c8c23bbc" exitCode=0
Nov 02 15:30:28.548709 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.548557    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns" event={"ID":"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5","Type":"ContainerDied","Data":"4b4493842e2c74ea36beefb3068ca2b3dd6a48191c0de8e86370e148c8c23bbc"}
Nov 02 15:30:28.548709 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.548705    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"
Nov 02 15:30:28.710776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.710688    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"]
Nov 02 15:30:28.714943 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.714917    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-s8wg7"]
Nov 02 15:30:28.794652 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.794425    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"]
Nov 02 15:30:28.794888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.794687    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" containerName="istio-proxy" containerID="cri-o://3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669" gracePeriod=30
Nov 02 15:30:28.798931 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.798905    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"]
Nov 02 15:30:28.799102 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.799084    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy" containerID="cri-o://5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7" gracePeriod=30
Nov 02 15:30:28.802440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.802415    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"]
Nov 02 15:30:28.802620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.802587    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy" containerID="cri-o://2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8" gracePeriod=30
Nov 02 15:30:28.817476 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.817448    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"]
Nov 02 15:30:28.817663 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.817642    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy" containerID="cri-o://9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8" gracePeriod=30
Nov 02 15:30:28.825355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.825331    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"]
Nov 02 15:30:28.844982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.844956    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-hzb6r"]
Nov 02 15:30:28.912605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:28.912575    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"]
Nov 02 15:30:29.557899 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.557865    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq" event={"ID":"cf99298c-7016-4838-98eb-f729a6d2398d","Type":"ContainerStarted","Data":"062f37198be3ab9ade81a94477bf720988c5460dff0d720f497b0e264d671fdd"}
Nov 02 15:30:29.558362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.557919    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq" event={"ID":"cf99298c-7016-4838-98eb-f729a6d2398d","Type":"ContainerStarted","Data":"5d0874ddbb17a729b5ed3804c75cdef1baa63c1089ad4f6960bbec5cd571457b"}
Nov 02 15:30:29.674956 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.674904    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq" podStartSLOduration=1.6748895560000001 podStartE2EDuration="1.674889556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:29.67247186 +0000 UTC m=+2037.265088660" watchObservedRunningTime="2026-11-02 15:30:29.674889556 +0000 UTC m=+2037.267506354"
Nov 02 15:30:29.711383 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.711360    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:29.894068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.893977    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-kube-api-access-ps4mj\" (UniqueName: \"kubernetes.io/projected/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-kube-api-access-ps4mj\") pod \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\" (UID: \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\") "
Nov 02 15:30:29.894240 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.894103    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-xks-ca\") pod \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\" (UID: \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\") "
Nov 02 15:30:29.894240 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.894134    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-openshift-ca\") pod \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\" (UID: \"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5\") "
Nov 02 15:30:29.894607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.894574    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-openshift-ca" pod "7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5" (UID: "7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:29.896145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.896120    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-kube-api-access-ps4mj" pod "7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5" (UID: "7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5"). InnerVolumeSpecName "kube-api-access-ps4mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:29.896269 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.896246    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-xks-ca" pod "7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5" (UID: "7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:30:29.995715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.995676    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:29.995866 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.995728    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:29.995866 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:29.995739    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps4mj\" (UniqueName: \"kubernetes.io/projected/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5-kube-api-access-ps4mj\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:30.523815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:30.523772    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7390b194-bcc2-4020-98c4-73cfb8d07480" path="/var/lib/kubelet/pods/7390b194-bcc2-4020-98c4-73cfb8d07480/volumes"
Nov 02 15:30:30.524350 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:30.524330    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ecc4598-19ce-463b-a3de-5ee6ba41a08f" path="/var/lib/kubelet/pods/8ecc4598-19ce-463b-a3de-5ee6ba41a08f/volumes"
Nov 02 15:30:30.568192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:30.568157    4891 generic.go:356] "Generic (PLEG): container finished" podID="cf99298c-7016-4838-98eb-f729a6d2398d" containerID="062f37198be3ab9ade81a94477bf720988c5460dff0d720f497b0e264d671fdd" exitCode=0
Nov 02 15:30:30.568683 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:30.568242    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq" event={"ID":"cf99298c-7016-4838-98eb-f729a6d2398d","Type":"ContainerDied","Data":"062f37198be3ab9ade81a94477bf720988c5460dff0d720f497b0e264d671fdd"}
Nov 02 15:30:30.569853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:30.569834    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"
Nov 02 15:30:30.569959 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:30.569832    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns" event={"ID":"7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5","Type":"ContainerDied","Data":"7a1b68869141452dbb8d6752850e57fdee7658434dd18d7e191fc0f7a8b9aa61"}
Nov 02 15:30:30.569959 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:30.569948    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a1b68869141452dbb8d6752850e57fdee7658434dd18d7e191fc0f7a8b9aa61"
Nov 02 15:30:31.740575 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:31.740546    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:31.920528 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:31.920444    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf99298c-7016-4838-98eb-f729a6d2398d-kube-api-access-vjf7q\" (UniqueName: \"kubernetes.io/projected/cf99298c-7016-4838-98eb-f729a6d2398d-kube-api-access-vjf7q\") pod \"cf99298c-7016-4838-98eb-f729a6d2398d\" (UID: \"cf99298c-7016-4838-98eb-f729a6d2398d\") "
Nov 02 15:30:31.920684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:31.920527    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-xks-ca\") pod \"cf99298c-7016-4838-98eb-f729a6d2398d\" (UID: \"cf99298c-7016-4838-98eb-f729a6d2398d\") "
Nov 02 15:30:31.920684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:31.920638    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-openshift-ca\") pod \"cf99298c-7016-4838-98eb-f729a6d2398d\" (UID: \"cf99298c-7016-4838-98eb-f729a6d2398d\") "
Nov 02 15:30:31.921008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:31.920962    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-openshift-ca" pod "cf99298c-7016-4838-98eb-f729a6d2398d" (UID: "cf99298c-7016-4838-98eb-f729a6d2398d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:31.922547 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:31.922522    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-xks-ca" pod "cf99298c-7016-4838-98eb-f729a6d2398d" (UID: "cf99298c-7016-4838-98eb-f729a6d2398d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:30:31.922664 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:31.922644    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf99298c-7016-4838-98eb-f729a6d2398d-kube-api-access-vjf7q" pod "cf99298c-7016-4838-98eb-f729a6d2398d" (UID: "cf99298c-7016-4838-98eb-f729a6d2398d"). InnerVolumeSpecName "kube-api-access-vjf7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:32.022586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.022553    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:32.022586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.022581    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjf7q\" (UniqueName: \"kubernetes.io/projected/cf99298c-7016-4838-98eb-f729a6d2398d-kube-api-access-vjf7q\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:32.022586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.022591    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf99298c-7016-4838-98eb-f729a6d2398d-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:32.304756 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.304711    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"]
Nov 02 15:30:32.305529 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.305507    4891 cpu_manager.go:414] "containerMap: removing container" podUID="cf99298c-7016-4838-98eb-f729a6d2398d" containerName="revoke-keys"
Nov 02 15:30:32.305578 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.305536    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf99298c-7016-4838-98eb-f729a6d2398d" containerName="revoke-keys"
Nov 02 15:30:32.305578 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.305549    4891 cpu_manager.go:414] "containerMap: removing container" podUID="7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5" containerName="revoke-keys"
Nov 02 15:30:32.305578 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.305555    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5" containerName="revoke-keys"
Nov 02 15:30:32.305692 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.305656    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf99298c-7016-4838-98eb-f729a6d2398d" containerName="revoke-keys"
Nov 02 15:30:32.305692 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.305665    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5" containerName="revoke-keys"
Nov 02 15:30:32.309055 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.309038    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:32.362865 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.362833    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"]
Nov 02 15:30:32.367512 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.367487    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"]
Nov 02 15:30:32.367666 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.367599    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:32.426727 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.426685    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2337f56a-d2nrj\" (UID: \"749b1dac-26a8-41de-bfe9-89f861b7e10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:32.426892 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.426826    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfwfc\" (UniqueName: \"kubernetes.io/projected/749b1dac-26a8-41de-bfe9-89f861b7e10a-kube-api-access-mfwfc\") pod \"maas-api-revoke-sub-2337f56a-d2nrj\" (UID: \"749b1dac-26a8-41de-bfe9-89f861b7e10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:32.426971 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.426931    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2337f56a-d2nrj\" (UID: \"749b1dac-26a8-41de-bfe9-89f861b7e10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:32.429133 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.429097    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"]
Nov 02 15:30:32.527831 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.527795    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfwfc\" (UniqueName: \"kubernetes.io/projected/749b1dac-26a8-41de-bfe9-89f861b7e10a-kube-api-access-mfwfc\") pod \"maas-api-revoke-sub-2337f56a-d2nrj\" (UID: \"749b1dac-26a8-41de-bfe9-89f861b7e10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:32.528024 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.527847    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2337f56a-d2nrj\" (UID: \"749b1dac-26a8-41de-bfe9-89f861b7e10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:32.528024 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.527942    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-575229fb-pf5tt\" (UID: \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:32.528162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.528036    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-575229fb-pf5tt\" (UID: \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:32.528162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.528124    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrv68\" (UniqueName: \"kubernetes.io/projected/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-kube-api-access-mrv68\") pod \"maas-api-revoke-sub-575229fb-pf5tt\" (UID: \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:32.528162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.528154    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2337f56a-d2nrj\" (UID: \"749b1dac-26a8-41de-bfe9-89f861b7e10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:32.528649 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.528627    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2337f56a-d2nrj\" (UID: \"749b1dac-26a8-41de-bfe9-89f861b7e10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:32.530440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.530417    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2337f56a-d2nrj\" (UID: \"749b1dac-26a8-41de-bfe9-89f861b7e10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:32.589799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.589512    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfwfc\" (UniqueName: \"kubernetes.io/projected/749b1dac-26a8-41de-bfe9-89f861b7e10a-kube-api-access-mfwfc\") pod \"maas-api-revoke-sub-2337f56a-d2nrj\" (UID: \"749b1dac-26a8-41de-bfe9-89f861b7e10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:32.590222 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.590193    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq" event={"ID":"cf99298c-7016-4838-98eb-f729a6d2398d","Type":"ContainerDied","Data":"5d0874ddbb17a729b5ed3804c75cdef1baa63c1089ad4f6960bbec5cd571457b"}
Nov 02 15:30:32.590365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.590292    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d0874ddbb17a729b5ed3804c75cdef1baa63c1089ad4f6960bbec5cd571457b"
Nov 02 15:30:32.602876 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.602840    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"
Nov 02 15:30:32.614101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.614072    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"]
Nov 02 15:30:32.624198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.624166    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:32.628946 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.628921    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-575229fb-pf5tt\" (UID: \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:32.629041 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.628957    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-575229fb-pf5tt\" (UID: \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:32.629150 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.629124    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrv68\" (UniqueName: \"kubernetes.io/projected/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-kube-api-access-mrv68\") pod \"maas-api-revoke-sub-575229fb-pf5tt\" (UID: \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:32.629768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.629735    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-575229fb-pf5tt\" (UID: \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:32.631938 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.631914    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-575229fb-pf5tt\" (UID: \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:32.637066 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.637044    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"]
Nov 02 15:30:32.693590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.693173    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrv68\" (UniqueName: \"kubernetes.io/projected/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-kube-api-access-mrv68\") pod \"maas-api-revoke-sub-575229fb-pf5tt\" (UID: \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:32.987835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:32.987762    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:33.007916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:33.007888    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:30:33.008018 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:33.007939    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:30:33.010530 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:30:33.010501    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749b1dac_26a8_41de_bfe9_89f861b7e10a.slice/crio-bd5c75f016f79d428d32828f6afcff293773df1ddbf775f0c0ccbf1d38bd9606 WatchSource:0}: Error finding container bd5c75f016f79d428d32828f6afcff293773df1ddbf775f0c0ccbf1d38bd9606: Status 404 returned error can't find the container with id bd5c75f016f79d428d32828f6afcff293773df1ddbf775f0c0ccbf1d38bd9606
Nov 02 15:30:33.010911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:33.010889    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"]
Nov 02 15:30:33.377920 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:33.377893    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"]
Nov 02 15:30:33.378369 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:30:33.378346    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88f500a_1a7e_40fe_806d_b3b026ab2eb4.slice/crio-ca8ffa4adb5e84c61e046dd76e627d4556f1136bd324415054b7033833e190ce WatchSource:0}: Error finding container ca8ffa4adb5e84c61e046dd76e627d4556f1136bd324415054b7033833e190ce: Status 404 returned error can't find the container with id ca8ffa4adb5e84c61e046dd76e627d4556f1136bd324415054b7033833e190ce
Nov 02 15:30:33.599888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:33.599849    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj" event={"ID":"749b1dac-26a8-41de-bfe9-89f861b7e10a","Type":"ContainerStarted","Data":"2d83c98643edb1a15525bdb6af61265d1c4832dad0a9780c7674d425f08e2b06"}
Nov 02 15:30:33.600083 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:33.599914    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj" event={"ID":"749b1dac-26a8-41de-bfe9-89f861b7e10a","Type":"ContainerStarted","Data":"bd5c75f016f79d428d32828f6afcff293773df1ddbf775f0c0ccbf1d38bd9606"}
Nov 02 15:30:33.601257 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:33.601229    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt" event={"ID":"f88f500a-1a7e-40fe-806d-b3b026ab2eb4","Type":"ContainerStarted","Data":"ca8ffa4adb5e84c61e046dd76e627d4556f1136bd324415054b7033833e190ce"}
Nov 02 15:30:33.722015 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:33.721961    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj" podStartSLOduration=1.721945968 podStartE2EDuration="1.721945968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:33.717959418 +0000 UTC m=+2041.310576218" watchObservedRunningTime="2026-11-02 15:30:33.721945968 +0000 UTC m=+2041.314562766"
Nov 02 15:30:33.844919 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:33.844879    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6qcng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 15:30:33.845066 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:33.844951    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 15:30:34.126991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.126959    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:30:34.129162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.129137    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:30:34.152861 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.152824    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6483392d-cba3-48d0-a898-73d7369d3e20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6483392d-cba3-48d0-a898-73d7369d3e20-istiod-ca-cert\") pod \"6483392d-cba3-48d0-a898-73d7369d3e20\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") "
Nov 02 15:30:34.153051 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.152968    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-certs\") pod \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") "
Nov 02 15:30:34.153051 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.153014    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-credential-socket\") pod \"6483392d-cba3-48d0-a898-73d7369d3e20\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") "
Nov 02 15:30:34.153186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.153074    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-socket\") pod \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") "
Nov 02 15:30:34.153186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.153104    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-socket\") pod \"6483392d-cba3-48d0-a898-73d7369d3e20\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") "
Nov 02 15:30:34.153186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.153136    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istiod-ca-cert\") pod \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") "
Nov 02 15:30:34.153186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.153172    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-kube-api-access-r8s97\" (UniqueName: \"kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-kube-api-access-r8s97\") pod \"6483392d-cba3-48d0-a898-73d7369d3e20\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") "
Nov 02 15:30:34.153439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.153227    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-credential-socket\") pod \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") "
Nov 02 15:30:34.153439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.153270    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-token\" (UniqueName: \"kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-token\") pod \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") "
Nov 02 15:30:34.153850 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.153264    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-certs" pod "68fb2ba3-1e31-4db1-9b63-94a48bff0663" (UID: "68fb2ba3-1e31-4db1-9b63-94a48bff0663"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.153850 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.153294    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6483392d-cba3-48d0-a898-73d7369d3e20-istiod-ca-cert" pod "6483392d-cba3-48d0-a898-73d7369d3e20" (UID: "6483392d-cba3-48d0-a898-73d7369d3e20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:34.153850 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.153507    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-socket" pod "6483392d-cba3-48d0-a898-73d7369d3e20" (UID: "6483392d-cba3-48d0-a898-73d7369d3e20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.153850 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.153724    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-socket" pod "68fb2ba3-1e31-4db1-9b63-94a48bff0663" (UID: "68fb2ba3-1e31-4db1-9b63-94a48bff0663"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.154230 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.154186    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-credential-socket" pod "68fb2ba3-1e31-4db1-9b63-94a48bff0663" (UID: "68fb2ba3-1e31-4db1-9b63-94a48bff0663"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.154409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.154288    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-certs\") pod \"6483392d-cba3-48d0-a898-73d7369d3e20\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") "
Nov 02 15:30:34.154409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.154321    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-credential-socket" pod "6483392d-cba3-48d0-a898-73d7369d3e20" (UID: "6483392d-cba3-48d0-a898-73d7369d3e20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.154409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.154339    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-data\") pod \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") "
Nov 02 15:30:34.154409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.154394    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-kube-api-access-wwt29\" (UniqueName: \"kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-kube-api-access-wwt29\") pod \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") "
Nov 02 15:30:34.155025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.154450    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6483392d-cba3-48d0-a898-73d7369d3e20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6483392d-cba3-48d0-a898-73d7369d3e20-istio-podinfo\") pod \"6483392d-cba3-48d0-a898-73d7369d3e20\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") "
Nov 02 15:30:34.155025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.154573    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-envoy\") pod \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") "
Nov 02 15:30:34.155025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.154621    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-envoy\") pod \"6483392d-cba3-48d0-a898-73d7369d3e20\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") "
Nov 02 15:30:34.155025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.154654    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-podinfo\") pod \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\" (UID: \"68fb2ba3-1e31-4db1-9b63-94a48bff0663\") "
Nov 02 15:30:34.155025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.154751    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-data" pod "68fb2ba3-1e31-4db1-9b63-94a48bff0663" (UID: "68fb2ba3-1e31-4db1-9b63-94a48bff0663"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.155025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.154764    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-data\") pod \"6483392d-cba3-48d0-a898-73d7369d3e20\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") "
Nov 02 15:30:34.155025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.154928    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istiod-ca-cert" pod "68fb2ba3-1e31-4db1-9b63-94a48bff0663" (UID: "68fb2ba3-1e31-4db1-9b63-94a48bff0663"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:34.155610 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.155189    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-data" pod "6483392d-cba3-48d0-a898-73d7369d3e20" (UID: "6483392d-cba3-48d0-a898-73d7369d3e20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.155826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.155800    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-certs" pod "6483392d-cba3-48d0-a898-73d7369d3e20" (UID: "6483392d-cba3-48d0-a898-73d7369d3e20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.156013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.155993    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.156106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.156013    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6483392d-cba3-48d0-a898-73d7369d3e20-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.156106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.156030    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.156106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.156046    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.156106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.156060    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.156106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.156075    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.156106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.156088    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.156106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.156102    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.156521 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.156116    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.156521 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.156131    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.156640 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.156603    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-kube-api-access-r8s97" pod "6483392d-cba3-48d0-a898-73d7369d3e20" (UID: "6483392d-cba3-48d0-a898-73d7369d3e20"). InnerVolumeSpecName "kube-api-access-r8s97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:34.158692 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.158650    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-kube-api-access-wwt29" pod "68fb2ba3-1e31-4db1-9b63-94a48bff0663" (UID: "68fb2ba3-1e31-4db1-9b63-94a48bff0663"). InnerVolumeSpecName "kube-api-access-wwt29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:34.159249 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.159223    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-podinfo" pod "68fb2ba3-1e31-4db1-9b63-94a48bff0663" (UID: "68fb2ba3-1e31-4db1-9b63-94a48bff0663"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:30:34.168564 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.168532    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-envoy" pod "6483392d-cba3-48d0-a898-73d7369d3e20" (UID: "6483392d-cba3-48d0-a898-73d7369d3e20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.168675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.168593    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-token" pod "68fb2ba3-1e31-4db1-9b63-94a48bff0663" (UID: "68fb2ba3-1e31-4db1-9b63-94a48bff0663"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:34.169204 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.169180    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-envoy" pod "68fb2ba3-1e31-4db1-9b63-94a48bff0663" (UID: "68fb2ba3-1e31-4db1-9b63-94a48bff0663"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.169294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.169177    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6483392d-cba3-48d0-a898-73d7369d3e20-istio-podinfo" pod "6483392d-cba3-48d0-a898-73d7369d3e20" (UID: "6483392d-cba3-48d0-a898-73d7369d3e20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:30:34.179218 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.179200    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:30:34.181922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.181906    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:30:34.256767 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.256732    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-podinfo\") pod \"adcd5628-e804-4d09-aa88-9af4a730ef2c\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") "
Nov 02 15:30:34.256967 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.256784    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-certs\") pod \"b3dc8955-8714-49cd-a3a3-001d46023fc3\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") "
Nov 02 15:30:34.256967 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.256814    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-token\" (UniqueName: \"kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-token\") pod \"adcd5628-e804-4d09-aa88-9af4a730ef2c\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") "
Nov 02 15:30:34.256967 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.256871    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-data\") pod \"b3dc8955-8714-49cd-a3a3-001d46023fc3\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") "
Nov 02 15:30:34.256967 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.256915    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-socket\") pod \"b3dc8955-8714-49cd-a3a3-001d46023fc3\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") "
Nov 02 15:30:34.256967 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.256948    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-podinfo\") pod \"b3dc8955-8714-49cd-a3a3-001d46023fc3\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") "
Nov 02 15:30:34.257258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.256982    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-envoy\") pod \"adcd5628-e804-4d09-aa88-9af4a730ef2c\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") "
Nov 02 15:30:34.257258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.257012    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adcd5628-e804-4d09-aa88-9af4a730ef2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcd5628-e804-4d09-aa88-9af4a730ef2c-istiod-ca-cert\") pod \"adcd5628-e804-4d09-aa88-9af4a730ef2c\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") "
Nov 02 15:30:34.257258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.257041    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-credential-socket\") pod \"adcd5628-e804-4d09-aa88-9af4a730ef2c\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") "
Nov 02 15:30:34.257258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.257076    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-kube-api-access-xhmq9\" (UniqueName: \"kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-kube-api-access-xhmq9\") pod \"b3dc8955-8714-49cd-a3a3-001d46023fc3\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") "
Nov 02 15:30:34.257258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.257122    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-certs\") pod \"adcd5628-e804-4d09-aa88-9af4a730ef2c\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") "
Nov 02 15:30:34.257258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.257151    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-envoy\") pod \"b3dc8955-8714-49cd-a3a3-001d46023fc3\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") "
Nov 02 15:30:34.257258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.257239    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-credential-socket\") pod \"b3dc8955-8714-49cd-a3a3-001d46023fc3\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") "
Nov 02 15:30:34.257676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.257274    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-socket\") pod \"adcd5628-e804-4d09-aa88-9af4a730ef2c\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") "
Nov 02 15:30:34.257676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.257345    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-data\") pod \"adcd5628-e804-4d09-aa88-9af4a730ef2c\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") "
Nov 02 15:30:34.257676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.257393    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-token\" (UniqueName: \"kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-token\") pod \"b3dc8955-8714-49cd-a3a3-001d46023fc3\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") "
Nov 02 15:30:34.257676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.257447    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-istio-token\" (UniqueName: \"kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-istio-token\") pod \"6483392d-cba3-48d0-a898-73d7369d3e20\" (UID: \"6483392d-cba3-48d0-a898-73d7369d3e20\") "
Nov 02 15:30:34.257676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.257539    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-kube-api-access-c48g6\" (UniqueName: \"kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-kube-api-access-c48g6\") pod \"adcd5628-e804-4d09-aa88-9af4a730ef2c\" (UID: \"adcd5628-e804-4d09-aa88-9af4a730ef2c\") "
Nov 02 15:30:34.257676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.257578    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3dc8955-8714-49cd-a3a3-001d46023fc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dc8955-8714-49cd-a3a3-001d46023fc3-istiod-ca-cert\") pod \"b3dc8955-8714-49cd-a3a3-001d46023fc3\" (UID: \"b3dc8955-8714-49cd-a3a3-001d46023fc3\") "
Nov 02 15:30:34.258395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.258183    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.258395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.258213    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6483392d-cba3-48d0-a898-73d7369d3e20-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.258395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.258229    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.258395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.258246    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8s97\" (UniqueName: \"kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-kube-api-access-r8s97\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.258395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.258264    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.258395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.258281    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwt29\" (UniqueName: \"kubernetes.io/projected/68fb2ba3-1e31-4db1-9b63-94a48bff0663-kube-api-access-wwt29\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.258395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.258298    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6483392d-cba3-48d0-a898-73d7369d3e20-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.258860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.258525    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-socket" pod "b3dc8955-8714-49cd-a3a3-001d46023fc3" (UID: "b3dc8955-8714-49cd-a3a3-001d46023fc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.258860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.258567    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-data" pod "adcd5628-e804-4d09-aa88-9af4a730ef2c" (UID: "adcd5628-e804-4d09-aa88-9af4a730ef2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.258860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.258771    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-data" pod "b3dc8955-8714-49cd-a3a3-001d46023fc3" (UID: "b3dc8955-8714-49cd-a3a3-001d46023fc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.259090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.259045    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-credential-socket" pod "adcd5628-e804-4d09-aa88-9af4a730ef2c" (UID: "adcd5628-e804-4d09-aa88-9af4a730ef2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.260675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.259486    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-podinfo" pod "adcd5628-e804-4d09-aa88-9af4a730ef2c" (UID: "adcd5628-e804-4d09-aa88-9af4a730ef2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:30:34.260675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.259743    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-certs" pod "b3dc8955-8714-49cd-a3a3-001d46023fc3" (UID: "b3dc8955-8714-49cd-a3a3-001d46023fc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.260675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.259780    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-certs" pod "adcd5628-e804-4d09-aa88-9af4a730ef2c" (UID: "adcd5628-e804-4d09-aa88-9af4a730ef2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.260675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.260107    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-kube-api-access-xhmq9" pod "b3dc8955-8714-49cd-a3a3-001d46023fc3" (UID: "b3dc8955-8714-49cd-a3a3-001d46023fc3"). InnerVolumeSpecName "kube-api-access-xhmq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:34.260675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.260188    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-credential-socket" pod "b3dc8955-8714-49cd-a3a3-001d46023fc3" (UID: "b3dc8955-8714-49cd-a3a3-001d46023fc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.260675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.260237    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-socket" pod "adcd5628-e804-4d09-aa88-9af4a730ef2c" (UID: "adcd5628-e804-4d09-aa88-9af4a730ef2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.260675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.260589    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dc8955-8714-49cd-a3a3-001d46023fc3-istiod-ca-cert" pod "b3dc8955-8714-49cd-a3a3-001d46023fc3" (UID: "b3dc8955-8714-49cd-a3a3-001d46023fc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:34.262222 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.261265    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-envoy" pod "adcd5628-e804-4d09-aa88-9af4a730ef2c" (UID: "adcd5628-e804-4d09-aa88-9af4a730ef2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.262222 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.261437    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-envoy" pod "b3dc8955-8714-49cd-a3a3-001d46023fc3" (UID: "b3dc8955-8714-49cd-a3a3-001d46023fc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:34.262222 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.261647    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcd5628-e804-4d09-aa88-9af4a730ef2c-istiod-ca-cert" pod "adcd5628-e804-4d09-aa88-9af4a730ef2c" (UID: "adcd5628-e804-4d09-aa88-9af4a730ef2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:34.263007 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.262964    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-podinfo" pod "b3dc8955-8714-49cd-a3a3-001d46023fc3" (UID: "b3dc8955-8714-49cd-a3a3-001d46023fc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:30:34.263135 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.263038    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-token" pod "b3dc8955-8714-49cd-a3a3-001d46023fc3" (UID: "b3dc8955-8714-49cd-a3a3-001d46023fc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:34.263372 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.263347    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-token" pod "adcd5628-e804-4d09-aa88-9af4a730ef2c" (UID: "adcd5628-e804-4d09-aa88-9af4a730ef2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:34.263531 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.263440    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-kube-api-access-c48g6" pod "adcd5628-e804-4d09-aa88-9af4a730ef2c" (UID: "adcd5628-e804-4d09-aa88-9af4a730ef2c"). InnerVolumeSpecName "kube-api-access-c48g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:34.264017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.263991    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-istio-token" pod "6483392d-cba3-48d0-a898-73d7369d3e20" (UID: "6483392d-cba3-48d0-a898-73d7369d3e20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:34.359937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359868    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dc8955-8714-49cd-a3a3-001d46023fc3-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.359937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359893    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.359937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359902    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.359937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359910    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.359937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359921    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.359937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359929    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.359937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359937    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.360407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359945    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.360407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359953    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcd5628-e804-4d09-aa88-9af4a730ef2c-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.360407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359961    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.360407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359970    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhmq9\" (UniqueName: \"kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-kube-api-access-xhmq9\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.360407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359980    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.360407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359987    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.360407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.359995    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc8955-8714-49cd-a3a3-001d46023fc3-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.360407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.360004    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.360407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.360012    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcd5628-e804-4d09-aa88-9af4a730ef2c-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.360407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.360019    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dc8955-8714-49cd-a3a3-001d46023fc3-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.360407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.360027    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6483392d-cba3-48d0-a898-73d7369d3e20-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.360407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.360035    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c48g6\" (UniqueName: \"kubernetes.io/projected/adcd5628-e804-4d09-aa88-9af4a730ef2c-kube-api-access-c48g6\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:34.612385 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.612247    4891 generic.go:356] "Generic (PLEG): container finished" podID="f88f500a-1a7e-40fe-806d-b3b026ab2eb4" containerID="7deb9d8a695c7d22f70de72e7d230c8c477b66a66dcd2e42ec1d3645d5ad4368" exitCode=0
Nov 02 15:30:34.612385 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.612346    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt" event={"ID":"f88f500a-1a7e-40fe-806d-b3b026ab2eb4","Type":"ContainerDied","Data":"7deb9d8a695c7d22f70de72e7d230c8c477b66a66dcd2e42ec1d3645d5ad4368"}
Nov 02 15:30:34.614378 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.614353    4891 generic.go:356] "Generic (PLEG): container finished" podID="749b1dac-26a8-41de-bfe9-89f861b7e10a" containerID="2d83c98643edb1a15525bdb6af61265d1c4832dad0a9780c7674d425f08e2b06" exitCode=0
Nov 02 15:30:34.614520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.614441    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj" event={"ID":"749b1dac-26a8-41de-bfe9-89f861b7e10a","Type":"ContainerDied","Data":"2d83c98643edb1a15525bdb6af61265d1c4832dad0a9780c7674d425f08e2b06"}
Nov 02 15:30:34.616098 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.616074    4891 generic.go:356] "Generic (PLEG): container finished" podID="6483392d-cba3-48d0-a898-73d7369d3e20" containerID="3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669" exitCode=0
Nov 02 15:30:34.616214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.616144    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"
Nov 02 15:30:34.617127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.617104    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" event={"ID":"6483392d-cba3-48d0-a898-73d7369d3e20","Type":"ContainerDied","Data":"3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669"}
Nov 02 15:30:34.617568 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.617173    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf" event={"ID":"6483392d-cba3-48d0-a898-73d7369d3e20","Type":"ContainerDied","Data":"f45975bad25426174ed78ba7bfac80c0990104186c322b0fdb0587e5fbcef8ea"}
Nov 02 15:30:34.617568 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.617208    4891 scope.go:122] "RemoveContainer" containerID="3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669"
Nov 02 15:30:34.618968 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.618125    4891 generic.go:356] "Generic (PLEG): container finished" podID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerID="2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8" exitCode=0
Nov 02 15:30:34.618968 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.618222    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" event={"ID":"b3dc8955-8714-49cd-a3a3-001d46023fc3","Type":"ContainerDied","Data":"2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8"}
Nov 02 15:30:34.618968 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.618259    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"
Nov 02 15:30:34.618968 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.618261    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt" event={"ID":"b3dc8955-8714-49cd-a3a3-001d46023fc3","Type":"ContainerDied","Data":"e56336e3fd0bcddb1b1c2ecbdd2b8c32a0ba9c182f56abab407a7738d0b100d9"}
Nov 02 15:30:34.620857 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.620834    4891 generic.go:356] "Generic (PLEG): container finished" podID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerID="9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8" exitCode=0
Nov 02 15:30:34.620999 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.620913    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"
Nov 02 15:30:34.621143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.621107    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" event={"ID":"adcd5628-e804-4d09-aa88-9af4a730ef2c","Type":"ContainerDied","Data":"9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8"}
Nov 02 15:30:34.621287 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.621263    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" event={"ID":"adcd5628-e804-4d09-aa88-9af4a730ef2c","Type":"ContainerDied","Data":"0091a7ebb0b327094e15d5f65cbbbf8e52397142957ebdf7f0d0fb70059fea96"}
Nov 02 15:30:34.623501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.623480    4891 generic.go:356] "Generic (PLEG): container finished" podID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerID="5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7" exitCode=0
Nov 02 15:30:34.623629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.623506    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" event={"ID":"68fb2ba3-1e31-4db1-9b63-94a48bff0663","Type":"ContainerDied","Data":"5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7"}
Nov 02 15:30:34.623629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.623571    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng" event={"ID":"68fb2ba3-1e31-4db1-9b63-94a48bff0663","Type":"ContainerDied","Data":"267b45fda596d6b3ba1105c7f1dd404b03088fe9d401aceb3f8c92a83d9515f2"}
Nov 02 15:30:34.623629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.623531    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"
Nov 02 15:30:34.631541 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.631515    4891 scope.go:122] "RemoveContainer" containerID="3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669"
Nov 02 15:30:34.631865 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:34.631839    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669\": container with ID starting with 3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669 not found: ID does not exist" containerID="3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669"
Nov 02 15:30:34.631957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.631873    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669"} err="failed to get container status \"3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669\": rpc error: code = NotFound desc = could not find container \"3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669\": container with ID starting with 3e3719fa61702f632c355450cd10be43b8ba3e05f3c94b38ac51d13ec4a5c669 not found: ID does not exist"
Nov 02 15:30:34.631957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.631890    4891 scope.go:122] "RemoveContainer" containerID="2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8"
Nov 02 15:30:34.645271 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.645243    4891 scope.go:122] "RemoveContainer" containerID="2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8"
Nov 02 15:30:34.645557 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:34.645537    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8\": container with ID starting with 2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8 not found: ID does not exist" containerID="2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8"
Nov 02 15:30:34.645667 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.645567    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8"} err="failed to get container status \"2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8\": rpc error: code = NotFound desc = could not find container \"2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8\": container with ID starting with 2e37332864f6ade43de0388b0beceb5c52c83f312eac3293ebcc798a489b90e8 not found: ID does not exist"
Nov 02 15:30:34.645667 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.645587    4891 scope.go:122] "RemoveContainer" containerID="9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8"
Nov 02 15:30:34.655729 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.655539    4891 scope.go:122] "RemoveContainer" containerID="540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de"
Nov 02 15:30:34.665869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.665851    4891 scope.go:122] "RemoveContainer" containerID="9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8"
Nov 02 15:30:34.666165 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:34.666138    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8\": container with ID starting with 9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8 not found: ID does not exist" containerID="9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8"
Nov 02 15:30:34.666245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.666177    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8"} err="failed to get container status \"9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8\": rpc error: code = NotFound desc = could not find container \"9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8\": container with ID starting with 9f518cf16bda2bbd55416b22e51b4c07618e1980ce788b2f45adbb8ae6b972f8 not found: ID does not exist"
Nov 02 15:30:34.666245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.666205    4891 scope.go:122] "RemoveContainer" containerID="540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de"
Nov 02 15:30:34.666475 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:34.666457    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de\": container with ID starting with 540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de not found: ID does not exist" containerID="540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de"
Nov 02 15:30:34.666516 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.666482    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de"} err="failed to get container status \"540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de\": rpc error: code = NotFound desc = could not find container \"540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de\": container with ID starting with 540fd9a42a3beee4e7a341b7b809bfb5d30cb3d2c0532ef7ddd0dddd853144de not found: ID does not exist"
Nov 02 15:30:34.666516 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.666498    4891 scope.go:122] "RemoveContainer" containerID="5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7"
Nov 02 15:30:34.675960 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.675943    4891 scope.go:122] "RemoveContainer" containerID="5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7"
Nov 02 15:30:34.676245 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:34.676219    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7\": container with ID starting with 5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7 not found: ID does not exist" containerID="5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7"
Nov 02 15:30:34.676294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.676251    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7"} err="failed to get container status \"5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7\": rpc error: code = NotFound desc = could not find container \"5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7\": container with ID starting with 5873449be86215dc1ad0c56bd6ab3373ac0f20b25d841efafbc6d3827b817be7 not found: ID does not exist"
Nov 02 15:30:34.961781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.961637    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"]
Nov 02 15:30:34.968441 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.968412    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6qcng"]
Nov 02 15:30:34.969215 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.969185    4891 scope.go:122] "RemoveContainer" containerID="a258eafe36a27577be13f6cd49ce0a6632e27e31bcb1cc2cf08e8a550fe8fdc2"
Nov 02 15:30:34.980916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.980895    4891 scope.go:122] "RemoveContainer" containerID="a6258f31d5b8c44cef6544147a77c5a7f698a98d641160161fe786611fa704aa"
Nov 02 15:30:34.992952 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:34.992927    4891 scope.go:122] "RemoveContainer" containerID="84b625e3d4ddb407046a35ddbecbb18be2ef00c46abe04b81b1b2fa67fd573f3"
Nov 02 15:30:35.081695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.081664    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"]
Nov 02 15:30:35.086054 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.086028    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lcdkf"]
Nov 02 15:30:35.154537 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.154506    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:30:35.154863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.154565    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:30:35.194688 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.194654    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"]
Nov 02 15:30:35.198988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.198961    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zzbkt"]
Nov 02 15:30:35.309302 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.309265    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"]
Nov 02 15:30:35.316590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.316559    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwd8j"]
Nov 02 15:30:35.796668 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.796646    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:35.821062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.821038    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:35.877078 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.877043    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-kube-api-access-mrv68\" (UniqueName: \"kubernetes.io/projected/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-kube-api-access-mrv68\") pod \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\" (UID: \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\") "
Nov 02 15:30:35.877267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.877126    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-xks-ca\") pod \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\" (UID: \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\") "
Nov 02 15:30:35.877267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.877204    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-openshift-ca\") pod \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\" (UID: \"f88f500a-1a7e-40fe-806d-b3b026ab2eb4\") "
Nov 02 15:30:35.877267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.877258    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-xks-ca\") pod \"749b1dac-26a8-41de-bfe9-89f861b7e10a\" (UID: \"749b1dac-26a8-41de-bfe9-89f861b7e10a\") "
Nov 02 15:30:35.877484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.877305    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/749b1dac-26a8-41de-bfe9-89f861b7e10a-kube-api-access-mfwfc\" (UniqueName: \"kubernetes.io/projected/749b1dac-26a8-41de-bfe9-89f861b7e10a-kube-api-access-mfwfc\") pod \"749b1dac-26a8-41de-bfe9-89f861b7e10a\" (UID: \"749b1dac-26a8-41de-bfe9-89f861b7e10a\") "
Nov 02 15:30:35.877484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.877420    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-openshift-ca\") pod \"749b1dac-26a8-41de-bfe9-89f861b7e10a\" (UID: \"749b1dac-26a8-41de-bfe9-89f861b7e10a\") "
Nov 02 15:30:35.877640 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.877605    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-openshift-ca" pod "f88f500a-1a7e-40fe-806d-b3b026ab2eb4" (UID: "f88f500a-1a7e-40fe-806d-b3b026ab2eb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:35.877838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.877813    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-openshift-ca" pod "749b1dac-26a8-41de-bfe9-89f861b7e10a" (UID: "749b1dac-26a8-41de-bfe9-89f861b7e10a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:35.877994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.877976    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:35.878058 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.878008    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:35.879339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.879293    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-kube-api-access-mrv68" pod "f88f500a-1a7e-40fe-806d-b3b026ab2eb4" (UID: "f88f500a-1a7e-40fe-806d-b3b026ab2eb4"). InnerVolumeSpecName "kube-api-access-mrv68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:35.879441 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.879406    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-xks-ca" pod "749b1dac-26a8-41de-bfe9-89f861b7e10a" (UID: "749b1dac-26a8-41de-bfe9-89f861b7e10a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:30:35.879712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.879697    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749b1dac-26a8-41de-bfe9-89f861b7e10a-kube-api-access-mfwfc" pod "749b1dac-26a8-41de-bfe9-89f861b7e10a" (UID: "749b1dac-26a8-41de-bfe9-89f861b7e10a"). InnerVolumeSpecName "kube-api-access-mfwfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:35.879757 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.879709    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-xks-ca" pod "f88f500a-1a7e-40fe-806d-b3b026ab2eb4" (UID: "f88f500a-1a7e-40fe-806d-b3b026ab2eb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:30:35.978836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.978799    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrv68\" (UniqueName: \"kubernetes.io/projected/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-kube-api-access-mrv68\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:35.978836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.978829    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88f500a-1a7e-40fe-806d-b3b026ab2eb4-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:35.978836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.978840    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/749b1dac-26a8-41de-bfe9-89f861b7e10a-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:35.979051 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:35.978848    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfwfc\" (UniqueName: \"kubernetes.io/projected/749b1dac-26a8-41de-bfe9-89f861b7e10a-kube-api-access-mfwfc\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:36.522462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:36.522430    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" path="/var/lib/kubelet/pods/6483392d-cba3-48d0-a898-73d7369d3e20/volumes"
Nov 02 15:30:36.522978 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:36.522963    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" path="/var/lib/kubelet/pods/68fb2ba3-1e31-4db1-9b63-94a48bff0663/volumes"
Nov 02 15:30:36.523453 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:36.523439    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" path="/var/lib/kubelet/pods/adcd5628-e804-4d09-aa88-9af4a730ef2c/volumes"
Nov 02 15:30:36.524254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:36.524229    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" path="/var/lib/kubelet/pods/b3dc8955-8714-49cd-a3a3-001d46023fc3/volumes"
Nov 02 15:30:36.640572 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:36.640539    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"
Nov 02 15:30:36.640766 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:36.640576    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt" event={"ID":"f88f500a-1a7e-40fe-806d-b3b026ab2eb4","Type":"ContainerDied","Data":"ca8ffa4adb5e84c61e046dd76e627d4556f1136bd324415054b7033833e190ce"}
Nov 02 15:30:36.640766 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:36.640626    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca8ffa4adb5e84c61e046dd76e627d4556f1136bd324415054b7033833e190ce"
Nov 02 15:30:36.642183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:36.642161    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"
Nov 02 15:30:36.642294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:36.642160    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj" event={"ID":"749b1dac-26a8-41de-bfe9-89f861b7e10a","Type":"ContainerDied","Data":"bd5c75f016f79d428d32828f6afcff293773df1ddbf775f0c0ccbf1d38bd9606"}
Nov 02 15:30:36.642294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:36.642286    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd5c75f016f79d428d32828f6afcff293773df1ddbf775f0c0ccbf1d38bd9606"
Nov 02 15:30:40.204031 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:40.203994    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"]
Nov 02 15:30:44.274086 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:44.274045    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:30:44.668098 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:44.668015    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"]
Nov 02 15:30:44.716640 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:44.716610    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-303ba8-747ns"]
Nov 02 15:30:45.095778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.095747    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"]
Nov 02 15:30:45.096254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096220    4891 cpu_manager.go:414] "containerMap: removing container" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy"
Nov 02 15:30:45.096254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096237    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy"
Nov 02 15:30:45.096254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096247    4891 cpu_manager.go:414] "containerMap: removing container" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy"
Nov 02 15:30:45.096254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096252    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy"
Nov 02 15:30:45.096470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096274    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f88f500a-1a7e-40fe-806d-b3b026ab2eb4" containerName="revoke-keys"
Nov 02 15:30:45.096470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096278    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f88f500a-1a7e-40fe-806d-b3b026ab2eb4" containerName="revoke-keys"
Nov 02 15:30:45.096470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096284    4891 cpu_manager.go:414] "containerMap: removing container" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" containerName="istio-proxy"
Nov 02 15:30:45.096470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096290    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" containerName="istio-proxy"
Nov 02 15:30:45.096470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096331    4891 cpu_manager.go:414] "containerMap: removing container" podUID="749b1dac-26a8-41de-bfe9-89f861b7e10a" containerName="revoke-keys"
Nov 02 15:30:45.096470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096338    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749b1dac-26a8-41de-bfe9-89f861b7e10a" containerName="revoke-keys"
Nov 02 15:30:45.096470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096351    4891 cpu_manager.go:414] "containerMap: removing container" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy"
Nov 02 15:30:45.096470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096358    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy"
Nov 02 15:30:45.096470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096451    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="f88f500a-1a7e-40fe-806d-b3b026ab2eb4" containerName="revoke-keys"
Nov 02 15:30:45.096470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096460    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="6483392d-cba3-48d0-a898-73d7369d3e20" containerName="istio-proxy"
Nov 02 15:30:45.096470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096469    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy"
Nov 02 15:30:45.096780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096479    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="749b1dac-26a8-41de-bfe9-89f861b7e10a" containerName="revoke-keys"
Nov 02 15:30:45.096780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096485    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dc8955-8714-49cd-a3a3-001d46023fc3" containerName="istio-proxy"
Nov 02 15:30:45.096780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096491    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy"
Nov 02 15:30:45.096780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.096496    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fb2ba3-1e31-4db1-9b63-94a48bff0663" containerName="istio-proxy"
Nov 02 15:30:45.101275 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.101256    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:45.158216 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.158183    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"]
Nov 02 15:30:45.189672 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.189629    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt\" (UID: \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:45.189833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.189743    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt\" (UID: \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:45.189897 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.189844    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbl8c\" (UniqueName: \"kubernetes.io/projected/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-kube-api-access-hbl8c\") pod \"maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt\" (UID: \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:45.202776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.202748    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:30:45.202956 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.202784    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:30:45.204857 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.204832    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc" podUID="90b5bb68-8249-4e62-a49e-8b85e4a706f1" containerName="payload-processing" containerID="cri-o://e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae" gracePeriod=25
Nov 02 15:30:45.248376 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.248338    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:30:45.248528 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.248437    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:30:45.290638 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.290607    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt\" (UID: \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:45.291011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.290672    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbl8c\" (UniqueName: \"kubernetes.io/projected/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-kube-api-access-hbl8c\") pod \"maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt\" (UID: \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:45.291011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.290797    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt\" (UID: \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:45.291446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.291413    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt\" (UID: \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:45.293006 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.292979    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt\" (UID: \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:45.351339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.351248    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbl8c\" (UniqueName: \"kubernetes.io/projected/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-kube-api-access-hbl8c\") pod \"maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt\" (UID: \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:45.415614 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.415581    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:45.802420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:45.802390    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"]
Nov 02 15:30:45.803476 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:30:45.803447    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da45e7e_015a_4afa_9e7a_ff48cc78f1a2.slice/crio-5f90edd2bfef1016a03d3d666901d78dc4b8912dcdf684fe4251bd03b8635403 WatchSource:0}: Error finding container 5f90edd2bfef1016a03d3d666901d78dc4b8912dcdf684fe4251bd03b8635403: Status 404 returned error can't find the container with id 5f90edd2bfef1016a03d3d666901d78dc4b8912dcdf684fe4251bd03b8635403
Nov 02 15:30:46.522178 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:46.522152    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5" path="/var/lib/kubelet/pods/7d7ee164-fd9d-4b6d-b4a1-9a30ff7095b5/volumes"
Nov 02 15:30:46.721937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:46.721889    4891 generic.go:356] "Generic (PLEG): container finished" podID="5da45e7e-015a-4afa-9e7a-ff48cc78f1a2" containerID="4df5f2ae70ac78c89a1944c3e685efe7c3e6ee403a2899ebb821416853c3ce6c" exitCode=0
Nov 02 15:30:46.722083 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:46.721962    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt" event={"ID":"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2","Type":"ContainerDied","Data":"4df5f2ae70ac78c89a1944c3e685efe7c3e6ee403a2899ebb821416853c3ce6c"}
Nov 02 15:30:46.722083 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:46.722012    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt" event={"ID":"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2","Type":"ContainerStarted","Data":"5f90edd2bfef1016a03d3d666901d78dc4b8912dcdf684fe4251bd03b8635403"}
Nov 02 15:30:47.638023 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:47.637978    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main" containerID="cri-o://b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57" gracePeriod=45
Nov 02 15:30:47.663209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:47.663174    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main" containerID="cri-o://6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31" gracePeriod=45
Nov 02 15:30:47.901722 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:47.900104    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:47.960410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:47.960384    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:30:47.963369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:47.963348    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:30:48.020520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.020459    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tls-certs\") pod \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") "
Nov 02 15:30:48.020520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.020508    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-kube-api-access-hbl8c\" (UniqueName: \"kubernetes.io/projected/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-kube-api-access-hbl8c\") pod \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\" (UID: \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\") "
Nov 02 15:30:48.020788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.020554    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f03b2d-75da-4e3a-8e75-5608d55a27f8-kube-api-access-2xjb7\" (UniqueName: \"kubernetes.io/projected/07f03b2d-75da-4e3a-8e75-5608d55a27f8-kube-api-access-2xjb7\") pod \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") "
Nov 02 15:30:48.020788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.020664    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-openshift-ca\") pod \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\" (UID: \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\") "
Nov 02 15:30:48.020788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.020710    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7b021f-cc78-4f81-9160-dedfde5efa2e-kube-api-access-q7s27\" (UniqueName: \"kubernetes.io/projected/ce7b021f-cc78-4f81-9160-dedfde5efa2e-kube-api-access-q7s27\") pod \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") "
Nov 02 15:30:48.020788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.020751    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tls-certs\") pod \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") "
Nov 02 15:30:48.021389 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.021158    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-xks-ca\") pod \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\" (UID: \"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2\") "
Nov 02 15:30:48.021389 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.021231    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-dshm\") pod \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") "
Nov 02 15:30:48.021389 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.021274    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tmp-dir\") pod \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") "
Nov 02 15:30:48.021389 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.021330    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-home\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-home\") pod \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") "
Nov 02 15:30:48.021657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.021400    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-model-cache\") pod \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") "
Nov 02 15:30:48.021657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.021431    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-model-cache\") pod \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") "
Nov 02 15:30:48.021657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.021482    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-home\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-home\") pod \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") "
Nov 02 15:30:48.021657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.021537    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-dshm\") pod \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\" (UID: \"ce7b021f-cc78-4f81-9160-dedfde5efa2e\") "
Nov 02 15:30:48.021657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.021580    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tmp-dir\") pod \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\" (UID: \"07f03b2d-75da-4e3a-8e75-5608d55a27f8\") "
Nov 02 15:30:48.021905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.021749    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-openshift-ca" pod "5da45e7e-015a-4afa-9e7a-ff48cc78f1a2" (UID: "5da45e7e-015a-4afa-9e7a-ff48cc78f1a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:48.022018 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.021991    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tmp-dir" pod "07f03b2d-75da-4e3a-8e75-5608d55a27f8" (UID: "07f03b2d-75da-4e3a-8e75-5608d55a27f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.022208 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.022191    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tmp-dir" pod "ce7b021f-cc78-4f81-9160-dedfde5efa2e" (UID: "ce7b021f-cc78-4f81-9160-dedfde5efa2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.022282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.022262    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.022357 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.022285    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.022413 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.022382    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-home" pod "ce7b021f-cc78-4f81-9160-dedfde5efa2e" (UID: "ce7b021f-cc78-4f81-9160-dedfde5efa2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.022596 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.022575    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-home" pod "07f03b2d-75da-4e3a-8e75-5608d55a27f8" (UID: "07f03b2d-75da-4e3a-8e75-5608d55a27f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.022982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.022634    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-model-cache" pod "ce7b021f-cc78-4f81-9160-dedfde5efa2e" (UID: "ce7b021f-cc78-4f81-9160-dedfde5efa2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.022982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.022788    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-model-cache" pod "07f03b2d-75da-4e3a-8e75-5608d55a27f8" (UID: "07f03b2d-75da-4e3a-8e75-5608d55a27f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.023948 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.023906    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tls-certs" pod "07f03b2d-75da-4e3a-8e75-5608d55a27f8" (UID: "07f03b2d-75da-4e3a-8e75-5608d55a27f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:30:48.024745 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.024718    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-kube-api-access-hbl8c" pod "5da45e7e-015a-4afa-9e7a-ff48cc78f1a2" (UID: "5da45e7e-015a-4afa-9e7a-ff48cc78f1a2"). InnerVolumeSpecName "kube-api-access-hbl8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:48.025233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.025149    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-xks-ca" pod "5da45e7e-015a-4afa-9e7a-ff48cc78f1a2" (UID: "5da45e7e-015a-4afa-9e7a-ff48cc78f1a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:30:48.025233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.025199    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tls-certs" pod "ce7b021f-cc78-4f81-9160-dedfde5efa2e" (UID: "ce7b021f-cc78-4f81-9160-dedfde5efa2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:30:48.025415 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.025269    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-dshm" pod "ce7b021f-cc78-4f81-9160-dedfde5efa2e" (UID: "ce7b021f-cc78-4f81-9160-dedfde5efa2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.025547 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.025528    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-dshm" pod "07f03b2d-75da-4e3a-8e75-5608d55a27f8" (UID: "07f03b2d-75da-4e3a-8e75-5608d55a27f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.025815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.025788    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f03b2d-75da-4e3a-8e75-5608d55a27f8-kube-api-access-2xjb7" pod "07f03b2d-75da-4e3a-8e75-5608d55a27f8" (UID: "07f03b2d-75da-4e3a-8e75-5608d55a27f8"). InnerVolumeSpecName "kube-api-access-2xjb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:48.026277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.026245    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7b021f-cc78-4f81-9160-dedfde5efa2e-kube-api-access-q7s27" pod "ce7b021f-cc78-4f81-9160-dedfde5efa2e" (UID: "ce7b021f-cc78-4f81-9160-dedfde5efa2e"). InnerVolumeSpecName "kube-api-access-q7s27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:48.123397 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123362    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.123397 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123395    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.123639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123406    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.123639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123415    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.123639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123426    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.123639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123439    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.123639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123450    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.123639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123458    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f03b2d-75da-4e3a-8e75-5608d55a27f8-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.123639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123466    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7b021f-cc78-4f81-9160-dedfde5efa2e-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.123639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123473    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f03b2d-75da-4e3a-8e75-5608d55a27f8-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.123639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123481    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbl8c\" (UniqueName: \"kubernetes.io/projected/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2-kube-api-access-hbl8c\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.123639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123491    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xjb7\" (UniqueName: \"kubernetes.io/projected/07f03b2d-75da-4e3a-8e75-5608d55a27f8-kube-api-access-2xjb7\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.123639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.123501    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7s27\" (UniqueName: \"kubernetes.io/projected/ce7b021f-cc78-4f81-9160-dedfde5efa2e-kube-api-access-q7s27\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:30:48.736958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.736922    4891 generic.go:356] "Generic (PLEG): container finished" podID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerID="6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31" exitCode=0
Nov 02 15:30:48.737447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.736972    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"
Nov 02 15:30:48.737447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.737000    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" event={"ID":"07f03b2d-75da-4e3a-8e75-5608d55a27f8","Type":"ContainerDied","Data":"6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31"}
Nov 02 15:30:48.737447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.737040    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz" event={"ID":"07f03b2d-75da-4e3a-8e75-5608d55a27f8","Type":"ContainerDied","Data":"ac407d29b76bca3a96dad63892c57fb146c51a409c9ff73091a11ca8cf31b08e"}
Nov 02 15:30:48.737447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.737065    4891 scope.go:122] "RemoveContainer" containerID="6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31"
Nov 02 15:30:48.738591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.738428    4891 generic.go:356] "Generic (PLEG): container finished" podID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerID="b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57" exitCode=0
Nov 02 15:30:48.738591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.738482    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"
Nov 02 15:30:48.738591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.738508    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" event={"ID":"ce7b021f-cc78-4f81-9160-dedfde5efa2e","Type":"ContainerDied","Data":"b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57"}
Nov 02 15:30:48.738591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.738545    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp" event={"ID":"ce7b021f-cc78-4f81-9160-dedfde5efa2e","Type":"ContainerDied","Data":"d51c820b91660736882fe02b8a9caec6810070d61b52591f18b9af9ed212a9e5"}
Nov 02 15:30:48.740267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.740252    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"
Nov 02 15:30:48.740433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.740246    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt" event={"ID":"5da45e7e-015a-4afa-9e7a-ff48cc78f1a2","Type":"ContainerDied","Data":"5f90edd2bfef1016a03d3d666901d78dc4b8912dcdf684fe4251bd03b8635403"}
Nov 02 15:30:48.740516 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.740465    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f90edd2bfef1016a03d3d666901d78dc4b8912dcdf684fe4251bd03b8635403"
Nov 02 15:30:48.748475 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.748456    4891 scope.go:122] "RemoveContainer" containerID="6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31"
Nov 02 15:30:48.748750 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:48.748730    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31\": container with ID starting with 6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31 not found: ID does not exist" containerID="6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31"
Nov 02 15:30:48.748825 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.748760    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31"} err="failed to get container status \"6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31\": rpc error: code = NotFound desc = could not find container \"6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31\": container with ID starting with 6f236a19f0e6b0209812ad5321b1be75cabc1c9e2533c11536ded3f103103f31 not found: ID does not exist"
Nov 02 15:30:48.748825 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.748779    4891 scope.go:122] "RemoveContainer" containerID="b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57"
Nov 02 15:30:48.758796 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.758772    4891 scope.go:122] "RemoveContainer" containerID="b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57"
Nov 02 15:30:48.759081 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:30:48.759058    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57\": container with ID starting with b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57 not found: ID does not exist" containerID="b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57"
Nov 02 15:30:48.759146 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.759095    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57"} err="failed to get container status \"b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57\": rpc error: code = NotFound desc = could not find container \"b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57\": container with ID starting with b6e1061f780cd199cc7fbf01bf773e2d37f4e67ea965020c5dce514b8921ee57 not found: ID does not exist"
Nov 02 15:30:48.903065 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.902981    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"]
Nov 02 15:30:48.906525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:48.906503    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/simulator-w4-6dc69b-kserve-75d48945f-fm4qz"]
Nov 02 15:30:49.079234 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:49.079130    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"]
Nov 02 15:30:49.079427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:49.079413    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6dc69b/distinct-2-w4-6dc69b-kserve-6696cc89b6-78mxp"]
Nov 02 15:30:49.316855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:49.316818    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc" podUID="90b5bb68-8249-4e62-a49e-8b85e4a706f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:30:50.524803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:50.524767    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" path="/var/lib/kubelet/pods/07f03b2d-75da-4e3a-8e75-5608d55a27f8/volumes"
Nov 02 15:30:50.525442 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:50.525421    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" path="/var/lib/kubelet/pods/ce7b021f-cc78-4f81-9160-dedfde5efa2e/volumes"
Nov 02 15:30:54.420508 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.420478    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"]
Nov 02 15:30:54.420970 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.420957    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main"
Nov 02 15:30:54.420970 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.420968    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main"
Nov 02 15:30:54.421045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.420994    4891 cpu_manager.go:414] "containerMap: removing container" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main"
Nov 02 15:30:54.421045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.421000    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main"
Nov 02 15:30:54.421045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.421013    4891 cpu_manager.go:414] "containerMap: removing container" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy"
Nov 02 15:30:54.421045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.421018    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adcd5628-e804-4d09-aa88-9af4a730ef2c" containerName="istio-proxy"
Nov 02 15:30:54.421045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.421026    4891 cpu_manager.go:414] "containerMap: removing container" podUID="5da45e7e-015a-4afa-9e7a-ff48cc78f1a2" containerName="revoke-keys"
Nov 02 15:30:54.421045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.421031    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da45e7e-015a-4afa-9e7a-ff48cc78f1a2" containerName="revoke-keys"
Nov 02 15:30:54.421237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.421106    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f03b2d-75da-4e3a-8e75-5608d55a27f8" containerName="main"
Nov 02 15:30:54.421237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.421118    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da45e7e-015a-4afa-9e7a-ff48cc78f1a2" containerName="revoke-keys"
Nov 02 15:30:54.421237 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.421127    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce7b021f-cc78-4f81-9160-dedfde5efa2e" containerName="main"
Nov 02 15:30:54.426222 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.426202    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.498831 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.498799    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-39098c-openshift-default-dockercfg-q7l8h\""
Nov 02 15:30:54.556468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.556405    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"]
Nov 02 15:30:54.601596 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.601561    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-envoy\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.601806 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.601599    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqjkh\" (UniqueName: \"kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-kube-api-access-wqjkh\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.601886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.601810    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-token\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.601943 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.601919    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-certs\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.601943 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.601941    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istiod-ca-cert\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.602061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.602011    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-credential-socket\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.602061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.602050    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-data\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.602204 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.602080    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-podinfo\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.602204 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.602165    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-socket\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.702985 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.702882    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-envoy\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.702985 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.702933    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqjkh\" (UniqueName: \"kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-kube-api-access-wqjkh\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.702985 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.702991    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-token\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.703285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.703050    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-certs\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.703285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.703077    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istiod-ca-cert\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.703285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.703126    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-credential-socket\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.703285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.703184    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-data\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.703536 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.703446    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-podinfo\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.703536 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.703487    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-socket\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.703536 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.703517    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-certs\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.703701 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.703599    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-data\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.703701 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.703622    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-credential-socket\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.703922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.703898    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istiod-ca-cert\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.703963 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.703945    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-socket\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.705304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.705274    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-envoy\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.705896 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.705877    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-podinfo\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.764649 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.764617    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-token\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:54.764796 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:54.764715    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqjkh\" (UniqueName: \"kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-kube-api-access-wqjkh\") pod \"e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") " pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:55.042869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:55.042828    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:55.480042 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:55.480011    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"]
Nov 02 15:30:55.481496 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:30:55.481302    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a444b7c_3af1_40a3_aa72_2932b17a8a54.slice/crio-bf6b42118a5ac696de10c26584c1d10d0626e5369c8d84c0d3b2c28bc74698e5 WatchSource:0}: Error finding container bf6b42118a5ac696de10c26584c1d10d0626e5369c8d84c0d3b2c28bc74698e5: Status 404 returned error can't find the container with id bf6b42118a5ac696de10c26584c1d10d0626e5369c8d84c0d3b2c28bc74698e5
Nov 02 15:30:55.484154 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:55.484108    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:30:55.484267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:55.484194    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:30:55.484267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:55.484229    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:30:55.803229 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:55.803196    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" event={"ID":"2a444b7c-3af1-40a3-aa72-2932b17a8a54","Type":"ContainerStarted","Data":"195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c"}
Nov 02 15:30:55.803456 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:55.803245    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" event={"ID":"2a444b7c-3af1-40a3-aa72-2932b17a8a54","Type":"ContainerStarted","Data":"bf6b42118a5ac696de10c26584c1d10d0626e5369c8d84c0d3b2c28bc74698e5"}
Nov 02 15:30:55.926213 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:55.926147    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" podStartSLOduration=1.926132314 podStartE2EDuration="1.926132314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:55.92271369 +0000 UTC m=+2063.515330511" watchObservedRunningTime="2026-11-02 15:30:55.926132314 +0000 UTC m=+2063.518749126"
Nov 02 15:30:56.043613 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:56.043573    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:56.053217 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:56.053187    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:56.810891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:56.810860    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:56.812009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:56.811985    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:30:59.317174 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:30:59.317138    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc" podUID="90b5bb68-8249-4e62-a49e-8b85e4a706f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:31:02.482497 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.482462    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"]
Nov 02 15:31:02.486984 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.486963    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"
Nov 02 15:31:02.543831 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.543799    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"]
Nov 02 15:31:02.589027 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.588984    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-39098c\""
Nov 02 15:31:02.589293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.588984    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-39098c-dockercfg-458qd\""
Nov 02 15:31:02.598539 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.598496    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7292\" (UniqueName: \"kubernetes.io/projected/cd686938-df40-4efe-ad36-53c6337f78e4-kube-api-access-q7292\") pod \"payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x\" (UID: \"cd686938-df40-4efe-ad36-53c6337f78e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"
Nov 02 15:31:02.598963 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.598934    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd686938-df40-4efe-ad36-53c6337f78e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x\" (UID: \"cd686938-df40-4efe-ad36-53c6337f78e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"
Nov 02 15:31:02.640752 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.640717    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"]
Nov 02 15:31:02.645187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.645160    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"
Nov 02 15:31:02.700535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.700500    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd686938-df40-4efe-ad36-53c6337f78e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x\" (UID: \"cd686938-df40-4efe-ad36-53c6337f78e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"
Nov 02 15:31:02.700745 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.700702    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7292\" (UniqueName: \"kubernetes.io/projected/cd686938-df40-4efe-ad36-53c6337f78e4-kube-api-access-q7292\") pod \"payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x\" (UID: \"cd686938-df40-4efe-ad36-53c6337f78e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"
Nov 02 15:31:02.702174 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.702148    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd686938-df40-4efe-ad36-53c6337f78e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x\" (UID: \"cd686938-df40-4efe-ad36-53c6337f78e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"
Nov 02 15:31:02.702994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.702969    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"]
Nov 02 15:31:02.762108 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.762030    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7292\" (UniqueName: \"kubernetes.io/projected/cd686938-df40-4efe-ad36-53c6337f78e4-kube-api-access-q7292\") pod \"payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x\" (UID: \"cd686938-df40-4efe-ad36-53c6337f78e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"
Nov 02 15:31:02.801688 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.801650    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"
Nov 02 15:31:02.802011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.801987    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e79f6e-44a4-4f84-a651-8963c8299b39-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26\" (UID: \"e8e79f6e-44a4-4f84-a651-8963c8299b39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"
Nov 02 15:31:02.802141 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.802124    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drmkx\" (UniqueName: \"kubernetes.io/projected/e8e79f6e-44a4-4f84-a651-8963c8299b39-kube-api-access-drmkx\") pod \"payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26\" (UID: \"e8e79f6e-44a4-4f84-a651-8963c8299b39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"
Nov 02 15:31:02.903553 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.903513    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e79f6e-44a4-4f84-a651-8963c8299b39-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26\" (UID: \"e8e79f6e-44a4-4f84-a651-8963c8299b39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"
Nov 02 15:31:02.903763 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.903663    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drmkx\" (UniqueName: \"kubernetes.io/projected/e8e79f6e-44a4-4f84-a651-8963c8299b39-kube-api-access-drmkx\") pod \"payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26\" (UID: \"e8e79f6e-44a4-4f84-a651-8963c8299b39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"
Nov 02 15:31:02.904800 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.904768    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e79f6e-44a4-4f84-a651-8963c8299b39-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26\" (UID: \"e8e79f6e-44a4-4f84-a651-8963c8299b39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"
Nov 02 15:31:02.965098 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.965058    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drmkx\" (UniqueName: \"kubernetes.io/projected/e8e79f6e-44a4-4f84-a651-8963c8299b39-kube-api-access-drmkx\") pod \"payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26\" (UID: \"e8e79f6e-44a4-4f84-a651-8963c8299b39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"
Nov 02 15:31:02.967688 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:02.967666    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"
Nov 02 15:31:03.188696 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:03.188667    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"]
Nov 02 15:31:03.189180 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:31:03.189156    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd686938_df40_4efe_ad36_53c6337f78e4.slice/crio-9bf83fb57d9cc8550840c92157ff2742587a75685993caa5c585395c82580468 WatchSource:0}: Error finding container 9bf83fb57d9cc8550840c92157ff2742587a75685993caa5c585395c82580468: Status 404 returned error can't find the container with id 9bf83fb57d9cc8550840c92157ff2742587a75685993caa5c585395c82580468
Nov 02 15:31:03.350803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:03.350771    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"]
Nov 02 15:31:03.351950 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:31:03.351921    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e79f6e_44a4_4f84_a651_8963c8299b39.slice/crio-59b7f547672466cda1b038bd417ece0447f3370de4b00699875a1734a4fd71de WatchSource:0}: Error finding container 59b7f547672466cda1b038bd417ece0447f3370de4b00699875a1734a4fd71de: Status 404 returned error can't find the container with id 59b7f547672466cda1b038bd417ece0447f3370de4b00699875a1734a4fd71de
Nov 02 15:31:03.861026 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:03.860983    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x" event={"ID":"cd686938-df40-4efe-ad36-53c6337f78e4","Type":"ContainerStarted","Data":"4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff"}
Nov 02 15:31:03.861529 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:03.861049    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x" event={"ID":"cd686938-df40-4efe-ad36-53c6337f78e4","Type":"ContainerStarted","Data":"9bf83fb57d9cc8550840c92157ff2742587a75685993caa5c585395c82580468"}
Nov 02 15:31:03.861529 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:03.861080    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"
Nov 02 15:31:03.862614 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:03.862588    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26" event={"ID":"e8e79f6e-44a4-4f84-a651-8963c8299b39","Type":"ContainerStarted","Data":"f7d58a6b3ef575c9ef5630f2460b101f9d7a544ed357599d8b8fc3394fcc68a7"}
Nov 02 15:31:03.862723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:03.862629    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26" event={"ID":"e8e79f6e-44a4-4f84-a651-8963c8299b39","Type":"ContainerStarted","Data":"59b7f547672466cda1b038bd417ece0447f3370de4b00699875a1734a4fd71de"}
Nov 02 15:31:03.862723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:03.862650    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"
Nov 02 15:31:03.979977 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:03.979920    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x" podStartSLOduration=1.9799048799999999 podStartE2EDuration="1.97990488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:03.976061524 +0000 UTC m=+2071.568678324" watchObservedRunningTime="2026-11-02 15:31:03.97990488 +0000 UTC m=+2071.572521690"
Nov 02 15:31:04.120849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:04.120747    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26" podStartSLOduration=2.120733337 podStartE2EDuration="2.120733337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:04.11746783 +0000 UTC m=+2071.710084635" watchObservedRunningTime="2026-11-02 15:31:04.120733337 +0000 UTC m=+2071.713350164"
Nov 02 15:31:05.493819 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.493792    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"
Nov 02 15:31:05.634384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.634353    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b5bb68-8249-4e62-a49e-8b85e4a706f1-kube-api-access-wlq6q\" (UniqueName: \"kubernetes.io/projected/90b5bb68-8249-4e62-a49e-8b85e4a706f1-kube-api-access-wlq6q\") pod \"90b5bb68-8249-4e62-a49e-8b85e4a706f1\" (UID: \"90b5bb68-8249-4e62-a49e-8b85e4a706f1\") "
Nov 02 15:31:05.634564 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.634401    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90b5bb68-8249-4e62-a49e-8b85e4a706f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b5bb68-8249-4e62-a49e-8b85e4a706f1-plugins-config-volume\") pod \"90b5bb68-8249-4e62-a49e-8b85e4a706f1\" (UID: \"90b5bb68-8249-4e62-a49e-8b85e4a706f1\") "
Nov 02 15:31:05.634858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.634826    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b5bb68-8249-4e62-a49e-8b85e4a706f1-plugins-config-volume" pod "90b5bb68-8249-4e62-a49e-8b85e4a706f1" (UID: "90b5bb68-8249-4e62-a49e-8b85e4a706f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:05.636679 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.636655    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b5bb68-8249-4e62-a49e-8b85e4a706f1-kube-api-access-wlq6q" pod "90b5bb68-8249-4e62-a49e-8b85e4a706f1" (UID: "90b5bb68-8249-4e62-a49e-8b85e4a706f1"). InnerVolumeSpecName "kube-api-access-wlq6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:05.736028 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.735946    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlq6q\" (UniqueName: \"kubernetes.io/projected/90b5bb68-8249-4e62-a49e-8b85e4a706f1-kube-api-access-wlq6q\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:05.736028 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.735979    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b5bb68-8249-4e62-a49e-8b85e4a706f1-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:05.877621 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.877589    4891 generic.go:356] "Generic (PLEG): container finished" podID="90b5bb68-8249-4e62-a49e-8b85e4a706f1" containerID="e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae" exitCode=0
Nov 02 15:31:05.877792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.877633    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"
Nov 02 15:31:05.877792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.877631    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc" event={"ID":"90b5bb68-8249-4e62-a49e-8b85e4a706f1","Type":"ContainerDied","Data":"e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae"}
Nov 02 15:31:05.877792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.877758    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc" event={"ID":"90b5bb68-8249-4e62-a49e-8b85e4a706f1","Type":"ContainerDied","Data":"f70d97361bdf98d5b0dfca428772e05dd94a45a52e06da6a5c8e3ab08dbcdcce"}
Nov 02 15:31:05.877792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.877789    4891 scope.go:122] "RemoveContainer" containerID="e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae"
Nov 02 15:31:05.888670 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.888651    4891 scope.go:122] "RemoveContainer" containerID="e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae"
Nov 02 15:31:05.888941 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:31:05.888922    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae\": container with ID starting with e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae not found: ID does not exist" containerID="e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae"
Nov 02 15:31:05.889000 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:05.888951    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae"} err="failed to get container status \"e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae\": rpc error: code = NotFound desc = could not find container \"e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae\": container with ID starting with e156d3fd3b70cf36bd59c5f309d70db7c930db4d69f72677ad1bc6112df7dfae not found: ID does not exist"
Nov 02 15:31:06.043699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:06.043652    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"]
Nov 02 15:31:06.047150 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:06.047125    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6dc69b-5fd6bbbf9f-gdvfc"]
Nov 02 15:31:06.522184 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:06.522149    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90b5bb68-8249-4e62-a49e-8b85e4a706f1" path="/var/lib/kubelet/pods/90b5bb68-8249-4e62-a49e-8b85e4a706f1/volumes"
Nov 02 15:31:11.224747 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:11.224703    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"]
Nov 02 15:31:11.225231 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:11.224950    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerName="istio-proxy" containerID="cri-o://cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d" gracePeriod=30
Nov 02 15:31:12.036010 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.035971    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"]
Nov 02 15:31:12.041706 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.041677    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-559973-v2ffr"]
Nov 02 15:31:12.392680 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.392606    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"]
Nov 02 15:31:12.393146 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.393133    4891 cpu_manager.go:414] "containerMap: removing container" podUID="90b5bb68-8249-4e62-a49e-8b85e4a706f1" containerName="payload-processing"
Nov 02 15:31:12.393193 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.393147    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b5bb68-8249-4e62-a49e-8b85e4a706f1" containerName="payload-processing"
Nov 02 15:31:12.393271 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.393262    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b5bb68-8249-4e62-a49e-8b85e4a706f1" containerName="payload-processing"
Nov 02 15:31:12.398062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.398040    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:12.453464 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.453430    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"]
Nov 02 15:31:12.497432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.497405    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:31:12.497432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.497410    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:31:12.514195 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.514152    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a1470ec-zdxvp\" (UID: \"7b7dde3a-91d5-435a-a513-87ba52e835e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:12.514388 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.514206    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdtzm\" (UniqueName: \"kubernetes.io/projected/7b7dde3a-91d5-435a-a513-87ba52e835e5-kube-api-access-fdtzm\") pod \"maas-api-revoke-sub-8a1470ec-zdxvp\" (UID: \"7b7dde3a-91d5-435a-a513-87ba52e835e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:12.514515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.514484    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a1470ec-zdxvp\" (UID: \"7b7dde3a-91d5-435a-a513-87ba52e835e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:12.523341 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.523299    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c53929f-ef7b-43d2-841c-d6eda17b5aeb" path="/var/lib/kubelet/pods/3c53929f-ef7b-43d2-841c-d6eda17b5aeb/volumes"
Nov 02 15:31:12.523828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.523807    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:31:12.523900 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.523837    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:31:12.615995 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.615952    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a1470ec-zdxvp\" (UID: \"7b7dde3a-91d5-435a-a513-87ba52e835e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:12.616190 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.616033    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a1470ec-zdxvp\" (UID: \"7b7dde3a-91d5-435a-a513-87ba52e835e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:12.616190 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.616058    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdtzm\" (UniqueName: \"kubernetes.io/projected/7b7dde3a-91d5-435a-a513-87ba52e835e5-kube-api-access-fdtzm\") pod \"maas-api-revoke-sub-8a1470ec-zdxvp\" (UID: \"7b7dde3a-91d5-435a-a513-87ba52e835e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:12.616793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.616770    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a1470ec-zdxvp\" (UID: \"7b7dde3a-91d5-435a-a513-87ba52e835e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:12.618413 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.618393    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a1470ec-zdxvp\" (UID: \"7b7dde3a-91d5-435a-a513-87ba52e835e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:12.677428 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.677346    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdtzm\" (UniqueName: \"kubernetes.io/projected/7b7dde3a-91d5-435a-a513-87ba52e835e5-kube-api-access-fdtzm\") pod \"maas-api-revoke-sub-8a1470ec-zdxvp\" (UID: \"7b7dde3a-91d5-435a-a513-87ba52e835e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:12.712115 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:12.712077    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:13.123795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:13.123766    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"]
Nov 02 15:31:13.124820 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:31:13.124788    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7dde3a_91d5_435a_a513_87ba52e835e5.slice/crio-cd93ad09e7092eb47866eabd1f8324fdc81011995d22c34670263aa9d7e5f1b9 WatchSource:0}: Error finding container cd93ad09e7092eb47866eabd1f8324fdc81011995d22c34670263aa9d7e5f1b9: Status 404 returned error can't find the container with id cd93ad09e7092eb47866eabd1f8324fdc81011995d22c34670263aa9d7e5f1b9
Nov 02 15:31:13.428238 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:13.428131    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"]
Nov 02 15:31:13.433893 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:13.433860    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-6b2f61-7l49b"]
Nov 02 15:31:13.987100 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:13.987060    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp" event={"ID":"7b7dde3a-91d5-435a-a513-87ba52e835e5","Type":"ContainerStarted","Data":"b9fcdd15c5ac21ab65f8b0b58b7554b57683d66f92c649ced1e087f755e9f185"}
Nov 02 15:31:13.987283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:13.987123    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp" event={"ID":"7b7dde3a-91d5-435a-a513-87ba52e835e5","Type":"ContainerStarted","Data":"cd93ad09e7092eb47866eabd1f8324fdc81011995d22c34670263aa9d7e5f1b9"}
Nov 02 15:31:14.104766 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:14.104714    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp" podStartSLOduration=2.104698825 podStartE2EDuration="2.104698825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:14.102014743 +0000 UTC m=+2081.694631539" watchObservedRunningTime="2026-11-02 15:31:14.104698825 +0000 UTC m=+2081.697315624"
Nov 02 15:31:14.274070 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:31:14.274030    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:31:14.522871 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:14.522839    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e34adf2-b639-4aee-863d-47cb29994a2e" path="/var/lib/kubelet/pods/1e34adf2-b639-4aee-863d-47cb29994a2e/volumes"
Nov 02 15:31:14.737770 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:14.737671    4891 patch_prober.go:29] interesting pod/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:31:14.737770 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:14.737740    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:31:14.872974 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:14.872944    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"
Nov 02 15:31:14.873135 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:14.873014    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"
Nov 02 15:31:14.996632 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:14.996543    4891 generic.go:356] "Generic (PLEG): container finished" podID="7b7dde3a-91d5-435a-a513-87ba52e835e5" containerID="b9fcdd15c5ac21ab65f8b0b58b7554b57683d66f92c649ced1e087f755e9f185" exitCode=0
Nov 02 15:31:14.996782 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:14.996632    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp" event={"ID":"7b7dde3a-91d5-435a-a513-87ba52e835e5","Type":"ContainerDied","Data":"b9fcdd15c5ac21ab65f8b0b58b7554b57683d66f92c649ced1e087f755e9f185"}
Nov 02 15:31:16.156046 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.156019    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:16.264239 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.264166    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-openshift-ca\") pod \"7b7dde3a-91d5-435a-a513-87ba52e835e5\" (UID: \"7b7dde3a-91d5-435a-a513-87ba52e835e5\") "
Nov 02 15:31:16.264384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.264261    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7dde3a-91d5-435a-a513-87ba52e835e5-kube-api-access-fdtzm\" (UniqueName: \"kubernetes.io/projected/7b7dde3a-91d5-435a-a513-87ba52e835e5-kube-api-access-fdtzm\") pod \"7b7dde3a-91d5-435a-a513-87ba52e835e5\" (UID: \"7b7dde3a-91d5-435a-a513-87ba52e835e5\") "
Nov 02 15:31:16.264439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.264418    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-xks-ca\") pod \"7b7dde3a-91d5-435a-a513-87ba52e835e5\" (UID: \"7b7dde3a-91d5-435a-a513-87ba52e835e5\") "
Nov 02 15:31:16.264576 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.264555    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-openshift-ca" pod "7b7dde3a-91d5-435a-a513-87ba52e835e5" (UID: "7b7dde3a-91d5-435a-a513-87ba52e835e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:16.264845 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.264826    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:16.266292 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.266267    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7dde3a-91d5-435a-a513-87ba52e835e5-kube-api-access-fdtzm" pod "7b7dde3a-91d5-435a-a513-87ba52e835e5" (UID: "7b7dde3a-91d5-435a-a513-87ba52e835e5"). InnerVolumeSpecName "kube-api-access-fdtzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:16.266410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.266285    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-xks-ca" pod "7b7dde3a-91d5-435a-a513-87ba52e835e5" (UID: "7b7dde3a-91d5-435a-a513-87ba52e835e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:31:16.365964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.365929    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdtzm\" (UniqueName: \"kubernetes.io/projected/7b7dde3a-91d5-435a-a513-87ba52e835e5-kube-api-access-fdtzm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:16.365964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.365960    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7dde3a-91d5-435a-a513-87ba52e835e5-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:16.497150 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.497122    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:31:16.568602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.568573    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-data\") pod \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") "
Nov 02 15:31:16.568777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.568637    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-kube-api-access-f752r\" (UniqueName: \"kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-kube-api-access-f752r\") pod \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") "
Nov 02 15:31:16.568777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.568681    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-podinfo\") pod \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") "
Nov 02 15:31:16.568777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.568706    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-certs\") pod \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") "
Nov 02 15:31:16.568777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.568754    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-credential-socket\") pod \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") "
Nov 02 15:31:16.568999 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.568809    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istiod-ca-cert\") pod \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") "
Nov 02 15:31:16.568999 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.568875    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-token\" (UniqueName: \"kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-token\") pod \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") "
Nov 02 15:31:16.568999 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.568890    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-data" pod "d7aa2fe0-c2d3-467e-938c-0379eedc35a5" (UID: "d7aa2fe0-c2d3-467e-938c-0379eedc35a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:16.568999 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.568960    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-envoy\") pod \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") "
Nov 02 15:31:16.569229 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.569001    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-socket\") pod \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\" (UID: \"d7aa2fe0-c2d3-467e-938c-0379eedc35a5\") "
Nov 02 15:31:16.569284 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.569219    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-credential-socket" pod "d7aa2fe0-c2d3-467e-938c-0379eedc35a5" (UID: "d7aa2fe0-c2d3-467e-938c-0379eedc35a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:16.569284 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.569236    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-certs" pod "d7aa2fe0-c2d3-467e-938c-0379eedc35a5" (UID: "d7aa2fe0-c2d3-467e-938c-0379eedc35a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:16.569476 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.569335    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istiod-ca-cert" pod "d7aa2fe0-c2d3-467e-938c-0379eedc35a5" (UID: "d7aa2fe0-c2d3-467e-938c-0379eedc35a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:16.569534 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.569471    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-socket" pod "d7aa2fe0-c2d3-467e-938c-0379eedc35a5" (UID: "d7aa2fe0-c2d3-467e-938c-0379eedc35a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:16.569684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.569665    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:16.569751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.569690    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:16.569751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.569705    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:16.569751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.569720    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:16.569751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.569733    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:16.570920 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.570893    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-podinfo" pod "d7aa2fe0-c2d3-467e-938c-0379eedc35a5" (UID: "d7aa2fe0-c2d3-467e-938c-0379eedc35a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:31:16.571034 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.570960    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-kube-api-access-f752r" pod "d7aa2fe0-c2d3-467e-938c-0379eedc35a5" (UID: "d7aa2fe0-c2d3-467e-938c-0379eedc35a5"). InnerVolumeSpecName "kube-api-access-f752r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:16.571564 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.571541    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-token" pod "d7aa2fe0-c2d3-467e-938c-0379eedc35a5" (UID: "d7aa2fe0-c2d3-467e-938c-0379eedc35a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:16.571635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.571537    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-envoy" pod "d7aa2fe0-c2d3-467e-938c-0379eedc35a5" (UID: "d7aa2fe0-c2d3-467e-938c-0379eedc35a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:16.670764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.670726    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:16.670764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.670758    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f752r\" (UniqueName: \"kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-kube-api-access-f752r\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:16.670764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.670770    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:16.671010 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:16.670779    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7aa2fe0-c2d3-467e-938c-0379eedc35a5-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:17.011823 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:17.011740    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"
Nov 02 15:31:17.011823 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:17.011759    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp" event={"ID":"7b7dde3a-91d5-435a-a513-87ba52e835e5","Type":"ContainerDied","Data":"cd93ad09e7092eb47866eabd1f8324fdc81011995d22c34670263aa9d7e5f1b9"}
Nov 02 15:31:17.011823 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:17.011808    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd93ad09e7092eb47866eabd1f8324fdc81011995d22c34670263aa9d7e5f1b9"
Nov 02 15:31:17.013231 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:17.013206    4891 generic.go:356] "Generic (PLEG): container finished" podID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerID="cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d" exitCode=0
Nov 02 15:31:17.013412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:17.013257    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"
Nov 02 15:31:17.013412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:17.013267    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" event={"ID":"d7aa2fe0-c2d3-467e-938c-0379eedc35a5","Type":"ContainerDied","Data":"cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d"}
Nov 02 15:31:17.013412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:17.013302    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw" event={"ID":"d7aa2fe0-c2d3-467e-938c-0379eedc35a5","Type":"ContainerDied","Data":"eb31f9d70c95923e6d60a8949e83b7c2790547c82dd6d5365d6c400a8c4d200f"}
Nov 02 15:31:17.013412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:17.013338    4891 scope.go:122] "RemoveContainer" containerID="cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d"
Nov 02 15:31:17.029396 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:17.029364    4891 scope.go:122] "RemoveContainer" containerID="cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d"
Nov 02 15:31:17.029759 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:31:17.029734    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d\": container with ID starting with cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d not found: ID does not exist" containerID="cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d"
Nov 02 15:31:17.029855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:17.029768    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d"} err="failed to get container status \"cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d\": rpc error: code = NotFound desc = could not find container \"cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d\": container with ID starting with cebe09ac661183bb6c6b02dbdc45bdc75cc2fa3081595e7ca77725289ddf2b2d not found: ID does not exist"
Nov 02 15:31:17.181096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:17.181058    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"]
Nov 02 15:31:17.207232 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:17.207192    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de35cd-openshift-default-684fbf96fc-kpchw"]
Nov 02 15:31:18.522450 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:18.522413    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" path="/var/lib/kubelet/pods/d7aa2fe0-c2d3-467e-938c-0379eedc35a5/volumes"
Nov 02 15:31:19.999931 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:19.999894    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"]
Nov 02 15:31:20.000638 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.000614    4891 cpu_manager.go:414] "containerMap: removing container" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerName="istio-proxy"
Nov 02 15:31:20.000745 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.000639    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerName="istio-proxy"
Nov 02 15:31:20.000745 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.000668    4891 cpu_manager.go:414] "containerMap: removing container" podUID="7b7dde3a-91d5-435a-a513-87ba52e835e5" containerName="revoke-keys"
Nov 02 15:31:20.000745 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.000675    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7dde3a-91d5-435a-a513-87ba52e835e5" containerName="revoke-keys"
Nov 02 15:31:20.000910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.000855    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7dde3a-91d5-435a-a513-87ba52e835e5" containerName="revoke-keys"
Nov 02 15:31:20.000910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.000883    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7aa2fe0-c2d3-467e-938c-0379eedc35a5" containerName="istio-proxy"
Nov 02 15:31:20.005900 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.005878    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:20.058557 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.058526    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:31:20.058965 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.058944    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:31:20.108199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.108168    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:31:20.108442 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.108427    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:31:20.109350 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.109330    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp946\" (UniqueName: \"kubernetes.io/projected/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-kube-api-access-jp946\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7\" (UID: \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:20.109462 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.109365    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7\" (UID: \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:20.109520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.109507    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7\" (UID: \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:20.112284 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.112262    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"]
Nov 02 15:31:20.210884 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.210850    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7\" (UID: \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:20.211089 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.210996    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7\" (UID: \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:20.211170 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.211107    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp946\" (UniqueName: \"kubernetes.io/projected/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-kube-api-access-jp946\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7\" (UID: \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:20.211749 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.211720    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7\" (UID: \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:20.213249 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.213226    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7\" (UID: \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:20.272099 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.272067    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp946\" (UniqueName: \"kubernetes.io/projected/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-kube-api-access-jp946\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7\" (UID: \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:20.320075 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.320037    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:20.703287 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:20.703255    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"]
Nov 02 15:31:20.704530 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:31:20.704506    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b1741a_c4c0_4e0a_819f_d9c4a6e60f9e.slice/crio-3ca4c3bbff75cf7ef2e330518bf710585f5725dea140f6d9c3ee59fbca53ff1c WatchSource:0}: Error finding container 3ca4c3bbff75cf7ef2e330518bf710585f5725dea140f6d9c3ee59fbca53ff1c: Status 404 returned error can't find the container with id 3ca4c3bbff75cf7ef2e330518bf710585f5725dea140f6d9c3ee59fbca53ff1c
Nov 02 15:31:21.043855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:21.043821    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7" event={"ID":"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e","Type":"ContainerStarted","Data":"21f696804bc9820140eb34cfe3cc64a353ab5d3e987a4e43df3a321b7f3aa695"}
Nov 02 15:31:21.044423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:21.043881    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7" event={"ID":"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e","Type":"ContainerStarted","Data":"3ca4c3bbff75cf7ef2e330518bf710585f5725dea140f6d9c3ee59fbca53ff1c"}
Nov 02 15:31:21.162586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:21.162517    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7" podStartSLOduration=2.162497949 podStartE2EDuration="2.162497949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:21.160814869 +0000 UTC m=+2088.753431684" watchObservedRunningTime="2026-11-02 15:31:21.162497949 +0000 UTC m=+2088.755114748"
Nov 02 15:31:21.855301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:21.855258    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"]
Nov 02 15:31:21.859487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:21.859465    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:21.918431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:21.918400    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"]
Nov 02 15:31:21.934083 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:21.934055    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5kfs\" (UniqueName: \"kubernetes.io/projected/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-kube-api-access-m5kfs\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp\" (UID: \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:21.934083 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:21.934087    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp\" (UID: \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:21.934354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:21.934116    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp\" (UID: \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:22.035122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:22.035088    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5kfs\" (UniqueName: \"kubernetes.io/projected/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-kube-api-access-m5kfs\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp\" (UID: \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:22.035122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:22.035126    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp\" (UID: \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:22.035429 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:22.035150    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp\" (UID: \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:22.035819 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:22.035792    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp\" (UID: \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:22.037569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:22.037544    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp\" (UID: \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:22.053849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:22.053818    4891 generic.go:356] "Generic (PLEG): container finished" podID="44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e" containerID="21f696804bc9820140eb34cfe3cc64a353ab5d3e987a4e43df3a321b7f3aa695" exitCode=0
Nov 02 15:31:22.054176 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:22.053898    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7" event={"ID":"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e","Type":"ContainerDied","Data":"21f696804bc9820140eb34cfe3cc64a353ab5d3e987a4e43df3a321b7f3aa695"}
Nov 02 15:31:22.097693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:22.097666    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5kfs\" (UniqueName: \"kubernetes.io/projected/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-kube-api-access-m5kfs\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp\" (UID: \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:22.173109 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:22.173009    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:22.558759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:22.558730    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"]
Nov 02 15:31:22.559756 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:31:22.559726    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba2ab6e_7dd4_4b12_8d0d_4fe378c22dd4.slice/crio-8fce0323f41e7e3c1a49f940426858179b2b3c6536cffe313983a22b267b8116 WatchSource:0}: Error finding container 8fce0323f41e7e3c1a49f940426858179b2b3c6536cffe313983a22b267b8116: Status 404 returned error can't find the container with id 8fce0323f41e7e3c1a49f940426858179b2b3c6536cffe313983a22b267b8116
Nov 02 15:31:23.062953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.062919    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp" event={"ID":"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4","Type":"ContainerStarted","Data":"761ab4d28830a6d9900aa83819268f8182cd3d64ce028a79db7503ca6b7f873d"}
Nov 02 15:31:23.063408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.062982    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp" event={"ID":"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4","Type":"ContainerStarted","Data":"8fce0323f41e7e3c1a49f940426858179b2b3c6536cffe313983a22b267b8116"}
Nov 02 15:31:23.184062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.183939    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp" podStartSLOduration=2.183925568 podStartE2EDuration="2.183925568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:23.181669444 +0000 UTC m=+2090.774286244" watchObservedRunningTime="2026-11-02 15:31:23.183925568 +0000 UTC m=+2090.776542367"
Nov 02 15:31:23.203419 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.203390    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:23.348020 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.347987    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-kube-api-access-jp946\" (UniqueName: \"kubernetes.io/projected/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-kube-api-access-jp946\") pod \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\" (UID: \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\") "
Nov 02 15:31:23.348193 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.348063    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-xks-ca\") pod \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\" (UID: \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\") "
Nov 02 15:31:23.348193 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.348121    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-openshift-ca\") pod \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\" (UID: \"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e\") "
Nov 02 15:31:23.348598 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.348563    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-openshift-ca" pod "44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e" (UID: "44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:23.348791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.348772    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:23.350499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.350460    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-xks-ca" pod "44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e" (UID: "44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:31:23.350615 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.350585    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-kube-api-access-jp946" pod "44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e" (UID: "44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e"). InnerVolumeSpecName "kube-api-access-jp946". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:23.449993 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.449958    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp946\" (UniqueName: \"kubernetes.io/projected/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-kube-api-access-jp946\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:23.449993 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:23.449993    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:24.071400 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:24.071355    4891 generic.go:356] "Generic (PLEG): container finished" podID="bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4" containerID="761ab4d28830a6d9900aa83819268f8182cd3d64ce028a79db7503ca6b7f873d" exitCode=0
Nov 02 15:31:24.071829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:24.071447    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp" event={"ID":"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4","Type":"ContainerDied","Data":"761ab4d28830a6d9900aa83819268f8182cd3d64ce028a79db7503ca6b7f873d"}
Nov 02 15:31:24.072942 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:24.072923    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"
Nov 02 15:31:24.073075 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:24.072968    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7" event={"ID":"44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e","Type":"ContainerDied","Data":"3ca4c3bbff75cf7ef2e330518bf710585f5725dea140f6d9c3ee59fbca53ff1c"}
Nov 02 15:31:24.073075 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:24.073011    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ca4c3bbff75cf7ef2e330518bf710585f5725dea140f6d9c3ee59fbca53ff1c"
Nov 02 15:31:25.224361 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:25.224336    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:25.370671 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:25.370578    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-kube-api-access-m5kfs\" (UniqueName: \"kubernetes.io/projected/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-kube-api-access-m5kfs\") pod \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\" (UID: \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\") "
Nov 02 15:31:25.370848 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:25.370681    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-openshift-ca\") pod \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\" (UID: \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\") "
Nov 02 15:31:25.370848 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:25.370710    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-xks-ca\") pod \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\" (UID: \"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4\") "
Nov 02 15:31:25.371084 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:25.371059    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-openshift-ca" pod "bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4" (UID: "bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:25.371261 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:25.371241    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:25.372851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:25.372817    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-kube-api-access-m5kfs" pod "bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4" (UID: "bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4"). InnerVolumeSpecName "kube-api-access-m5kfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:25.372933 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:25.372864    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-xks-ca" pod "bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4" (UID: "bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:31:25.471943 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:25.471911    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5kfs\" (UniqueName: \"kubernetes.io/projected/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-kube-api-access-m5kfs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:25.471943 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:25.471940    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:26.089255 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:26.089226    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"
Nov 02 15:31:26.089465 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:26.089264    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp" event={"ID":"bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4","Type":"ContainerDied","Data":"8fce0323f41e7e3c1a49f940426858179b2b3c6536cffe313983a22b267b8116"}
Nov 02 15:31:26.089465 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:26.089347    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fce0323f41e7e3c1a49f940426858179b2b3c6536cffe313983a22b267b8116"
Nov 02 15:31:33.174828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:33.174792    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:31:33.176862 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:33.176837    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:31:33.178380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:33.178356    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:31:33.179451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:33.179431    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:31:33.192511 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:33.192488    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:31:33.194656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:33.194633    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:31:34.244461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:34.244424    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"]
Nov 02 15:31:39.244953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:39.244888    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz" podUID="07719395-3bc0-4e9a-b84c-af507e734d1f" containerName="payload-pre-processing" containerID="cri-o://bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546" gracePeriod=25
Nov 02 15:31:39.527826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:39.527799    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"
Nov 02 15:31:39.631069 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:39.631024    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07719395-3bc0-4e9a-b84c-af507e734d1f-kube-api-access-xcltj\" (UniqueName: \"kubernetes.io/projected/07719395-3bc0-4e9a-b84c-af507e734d1f-kube-api-access-xcltj\") pod \"07719395-3bc0-4e9a-b84c-af507e734d1f\" (UID: \"07719395-3bc0-4e9a-b84c-af507e734d1f\") "
Nov 02 15:31:39.631271 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:39.631112    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07719395-3bc0-4e9a-b84c-af507e734d1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07719395-3bc0-4e9a-b84c-af507e734d1f-plugins-config-volume\") pod \"07719395-3bc0-4e9a-b84c-af507e734d1f\" (UID: \"07719395-3bc0-4e9a-b84c-af507e734d1f\") "
Nov 02 15:31:39.631552 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:39.631530    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07719395-3bc0-4e9a-b84c-af507e734d1f-plugins-config-volume" pod "07719395-3bc0-4e9a-b84c-af507e734d1f" (UID: "07719395-3bc0-4e9a-b84c-af507e734d1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:39.633295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:39.633272    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07719395-3bc0-4e9a-b84c-af507e734d1f-kube-api-access-xcltj" pod "07719395-3bc0-4e9a-b84c-af507e734d1f" (UID: "07719395-3bc0-4e9a-b84c-af507e734d1f"). InnerVolumeSpecName "kube-api-access-xcltj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:39.732420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:39.732386    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcltj\" (UniqueName: \"kubernetes.io/projected/07719395-3bc0-4e9a-b84c-af507e734d1f-kube-api-access-xcltj\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:39.732420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:39.732419    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07719395-3bc0-4e9a-b84c-af507e734d1f-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:40.191816 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:40.191782    4891 generic.go:356] "Generic (PLEG): container finished" podID="07719395-3bc0-4e9a-b84c-af507e734d1f" containerID="bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546" exitCode=0
Nov 02 15:31:40.191816 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:40.191821    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz" event={"ID":"07719395-3bc0-4e9a-b84c-af507e734d1f","Type":"ContainerDied","Data":"bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546"}
Nov 02 15:31:40.192054 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:40.191827    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"
Nov 02 15:31:40.192054 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:40.191863    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz" event={"ID":"07719395-3bc0-4e9a-b84c-af507e734d1f","Type":"ContainerDied","Data":"0cbf859ed48e2e21099b3d31db6a93a23c7014a9f380de783f5b12a818849294"}
Nov 02 15:31:40.192054 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:40.191887    4891 scope.go:122] "RemoveContainer" containerID="bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546"
Nov 02 15:31:40.202618 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:40.202595    4891 scope.go:122] "RemoveContainer" containerID="bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546"
Nov 02 15:31:40.202925 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:31:40.202907    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546\": container with ID starting with bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546 not found: ID does not exist" containerID="bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546"
Nov 02 15:31:40.202973 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:40.202934    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546"} err="failed to get container status \"bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546\": rpc error: code = NotFound desc = could not find container \"bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546\": container with ID starting with bfa4883a7474fbf60a08c5dca25b798a031da4a636b45388cee7e864c9427546 not found: ID does not exist"
Nov 02 15:31:40.358666 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:40.358633    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"]
Nov 02 15:31:40.362914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:40.362883    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-885c69-784df4fff-gn9bz"]
Nov 02 15:31:40.523598 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:40.523570    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07719395-3bc0-4e9a-b84c-af507e734d1f" path="/var/lib/kubelet/pods/07719395-3bc0-4e9a-b84c-af507e734d1f/volumes"
Nov 02 15:31:44.273759 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:31:44.273678    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:31:48.023733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.023696    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"]
Nov 02 15:31:48.024189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.024176    4891 cpu_manager.go:414] "containerMap: removing container" podUID="07719395-3bc0-4e9a-b84c-af507e734d1f" containerName="payload-pre-processing"
Nov 02 15:31:48.024236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.024190    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07719395-3bc0-4e9a-b84c-af507e734d1f" containerName="payload-pre-processing"
Nov 02 15:31:48.024236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.024202    4891 cpu_manager.go:414] "containerMap: removing container" podUID="44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e" containerName="revoke-keys"
Nov 02 15:31:48.024236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.024208    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e" containerName="revoke-keys"
Nov 02 15:31:48.024236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.024229    4891 cpu_manager.go:414] "containerMap: removing container" podUID="bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4" containerName="revoke-keys"
Nov 02 15:31:48.024236 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.024234    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4" containerName="revoke-keys"
Nov 02 15:31:48.024433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.024339    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4" containerName="revoke-keys"
Nov 02 15:31:48.024433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.024359    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="07719395-3bc0-4e9a-b84c-af507e734d1f" containerName="payload-pre-processing"
Nov 02 15:31:48.024433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.024369    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e" containerName="revoke-keys"
Nov 02 15:31:48.029081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.029060    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:48.031369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.031345    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"]
Nov 02 15:31:48.035760 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.035742    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:48.081349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.081281    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:31:48.081349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.081277    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:31:48.081606 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.081377    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:31:48.130620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.130588    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:31:48.135071 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.135043    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"]
Nov 02 15:31:48.138450 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.138425    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"]
Nov 02 15:31:48.141378 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.141350    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c0fa4d-q7xn6\" (UID: \"db1ae3e8-21e1-4283-8917-0600eafdddf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:48.141492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.141427    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frnkp\" (UniqueName: \"kubernetes.io/projected/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-kube-api-access-frnkp\") pod \"maas-api-revoke-sub-a086b6ea-54nqx\" (UID: \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:48.141572 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.141547    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c0fa4d-q7xn6\" (UID: \"db1ae3e8-21e1-4283-8917-0600eafdddf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:48.141631 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.141608    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a086b6ea-54nqx\" (UID: \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:48.141773 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.141755    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a086b6ea-54nqx\" (UID: \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:48.141853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.141837    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvzw5\" (UniqueName: \"kubernetes.io/projected/db1ae3e8-21e1-4283-8917-0600eafdddf5-kube-api-access-wvzw5\") pod \"maas-api-revoke-sub-e2c0fa4d-q7xn6\" (UID: \"db1ae3e8-21e1-4283-8917-0600eafdddf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:48.243286 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.243249    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c0fa4d-q7xn6\" (UID: \"db1ae3e8-21e1-4283-8917-0600eafdddf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:48.243538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.243307    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a086b6ea-54nqx\" (UID: \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:48.243538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.243416    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a086b6ea-54nqx\" (UID: \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:48.243538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.243488    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzw5\" (UniqueName: \"kubernetes.io/projected/db1ae3e8-21e1-4283-8917-0600eafdddf5-kube-api-access-wvzw5\") pod \"maas-api-revoke-sub-e2c0fa4d-q7xn6\" (UID: \"db1ae3e8-21e1-4283-8917-0600eafdddf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:48.243731 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.243544    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c0fa4d-q7xn6\" (UID: \"db1ae3e8-21e1-4283-8917-0600eafdddf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:48.243731 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.243603    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frnkp\" (UniqueName: \"kubernetes.io/projected/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-kube-api-access-frnkp\") pod \"maas-api-revoke-sub-a086b6ea-54nqx\" (UID: \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:48.244238 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.244208    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a086b6ea-54nqx\" (UID: \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:48.244423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.244222    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c0fa4d-q7xn6\" (UID: \"db1ae3e8-21e1-4283-8917-0600eafdddf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:48.246544 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.246519    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a086b6ea-54nqx\" (UID: \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:48.246675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.246519    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c0fa4d-q7xn6\" (UID: \"db1ae3e8-21e1-4283-8917-0600eafdddf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:48.304867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.304797    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzw5\" (UniqueName: \"kubernetes.io/projected/db1ae3e8-21e1-4283-8917-0600eafdddf5-kube-api-access-wvzw5\") pod \"maas-api-revoke-sub-e2c0fa4d-q7xn6\" (UID: \"db1ae3e8-21e1-4283-8917-0600eafdddf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:48.305006 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.304907    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frnkp\" (UniqueName: \"kubernetes.io/projected/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-kube-api-access-frnkp\") pod \"maas-api-revoke-sub-a086b6ea-54nqx\" (UID: \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:48.344612 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.344578    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:48.350882 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.350855    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:48.746846 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.746812    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"]
Nov 02 15:31:48.747806 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:31:48.747781    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b1efe6_5e7d_4324_8bcb_b80686cdfbb3.slice/crio-bad3fa3e7922d0c287e76db5e40143d342abffd6e90c130c459d97d141658735 WatchSource:0}: Error finding container bad3fa3e7922d0c287e76db5e40143d342abffd6e90c130c459d97d141658735: Status 404 returned error can't find the container with id bad3fa3e7922d0c287e76db5e40143d342abffd6e90c130c459d97d141658735
Nov 02 15:31:48.796726 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:48.796693    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"]
Nov 02 15:31:48.797710 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:31:48.797686    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1ae3e8_21e1_4283_8917_0600eafdddf5.slice/crio-367dbd0effe94c6f0cd7bc660a7569446f50d03b386e431583a76d7e82a62f36 WatchSource:0}: Error finding container 367dbd0effe94c6f0cd7bc660a7569446f50d03b386e431583a76d7e82a62f36: Status 404 returned error can't find the container with id 367dbd0effe94c6f0cd7bc660a7569446f50d03b386e431583a76d7e82a62f36
Nov 02 15:31:49.258378 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:49.258304    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx" event={"ID":"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3","Type":"ContainerStarted","Data":"e35cefaf59a65c078ba122c6d8ea445cecdc9fbd20defe2a8e3338a0d50c7524"}
Nov 02 15:31:49.258856 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:49.258404    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx" event={"ID":"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3","Type":"ContainerStarted","Data":"bad3fa3e7922d0c287e76db5e40143d342abffd6e90c130c459d97d141658735"}
Nov 02 15:31:49.260191 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:49.260161    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6" event={"ID":"db1ae3e8-21e1-4283-8917-0600eafdddf5","Type":"ContainerStarted","Data":"8f675769c6db7102ef7811d46d91212a0c42cbe4f8ef7d23d66d81ee9d682e90"}
Nov 02 15:31:49.260332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:49.260209    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6" event={"ID":"db1ae3e8-21e1-4283-8917-0600eafdddf5","Type":"ContainerStarted","Data":"367dbd0effe94c6f0cd7bc660a7569446f50d03b386e431583a76d7e82a62f36"}
Nov 02 15:31:49.375364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:49.375287    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx" podStartSLOduration=2.375270003 podStartE2EDuration="2.375270003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:49.37332865 +0000 UTC m=+2116.965945441" watchObservedRunningTime="2026-11-02 15:31:49.375270003 +0000 UTC m=+2116.967886802"
Nov 02 15:31:49.492559 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:49.492447    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6" podStartSLOduration=2.492433748 podStartE2EDuration="2.492433748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:49.488398017 +0000 UTC m=+2117.081014818" watchObservedRunningTime="2026-11-02 15:31:49.492433748 +0000 UTC m=+2117.085050565"
Nov 02 15:31:50.260401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:50.260303    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"]
Nov 02 15:31:50.269612 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:50.269575    4891 generic.go:356] "Generic (PLEG): container finished" podID="31b1efe6-5e7d-4324-8bcb-b80686cdfbb3" containerID="e35cefaf59a65c078ba122c6d8ea445cecdc9fbd20defe2a8e3338a0d50c7524" exitCode=0
Nov 02 15:31:50.269793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:50.269660    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx" event={"ID":"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3","Type":"ContainerDied","Data":"e35cefaf59a65c078ba122c6d8ea445cecdc9fbd20defe2a8e3338a0d50c7524"}
Nov 02 15:31:50.271609 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:50.271578    4891 generic.go:356] "Generic (PLEG): container finished" podID="db1ae3e8-21e1-4283-8917-0600eafdddf5" containerID="8f675769c6db7102ef7811d46d91212a0c42cbe4f8ef7d23d66d81ee9d682e90" exitCode=0
Nov 02 15:31:50.271738 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:50.271659    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6" event={"ID":"db1ae3e8-21e1-4283-8917-0600eafdddf5","Type":"ContainerDied","Data":"8f675769c6db7102ef7811d46d91212a0c42cbe4f8ef7d23d66d81ee9d682e90"}
Nov 02 15:31:50.310797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:50.310762    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ba49d0-c5rwd"]
Nov 02 15:31:50.528809 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:50.528740    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c376fb6b-a1be-4423-8510-163029504e1d" path="/var/lib/kubelet/pods/c376fb6b-a1be-4423-8510-163029504e1d/volumes"
Nov 02 15:31:51.471856 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.471830    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:51.476377 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.476353    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:51.587189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.587073    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-openshift-ca\") pod \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\" (UID: \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\") "
Nov 02 15:31:51.587427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.587188    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-xks-ca\") pod \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\" (UID: \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\") "
Nov 02 15:31:51.587427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.587263    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-xks-ca\") pod \"db1ae3e8-21e1-4283-8917-0600eafdddf5\" (UID: \"db1ae3e8-21e1-4283-8917-0600eafdddf5\") "
Nov 02 15:31:51.587427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.587362    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-openshift-ca\") pod \"db1ae3e8-21e1-4283-8917-0600eafdddf5\" (UID: \"db1ae3e8-21e1-4283-8917-0600eafdddf5\") "
Nov 02 15:31:51.587427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.587414    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-kube-api-access-frnkp\" (UniqueName: \"kubernetes.io/projected/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-kube-api-access-frnkp\") pod \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\" (UID: \"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3\") "
Nov 02 15:31:51.587733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.587464    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1ae3e8-21e1-4283-8917-0600eafdddf5-kube-api-access-wvzw5\" (UniqueName: \"kubernetes.io/projected/db1ae3e8-21e1-4283-8917-0600eafdddf5-kube-api-access-wvzw5\") pod \"db1ae3e8-21e1-4283-8917-0600eafdddf5\" (UID: \"db1ae3e8-21e1-4283-8917-0600eafdddf5\") "
Nov 02 15:31:51.587733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.587517    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-openshift-ca" pod "31b1efe6-5e7d-4324-8bcb-b80686cdfbb3" (UID: "31b1efe6-5e7d-4324-8bcb-b80686cdfbb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:51.588242 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.587945    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:51.588242 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.587943    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-openshift-ca" pod "db1ae3e8-21e1-4283-8917-0600eafdddf5" (UID: "db1ae3e8-21e1-4283-8917-0600eafdddf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:51.591838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.591797    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-kube-api-access-frnkp" pod "31b1efe6-5e7d-4324-8bcb-b80686cdfbb3" (UID: "31b1efe6-5e7d-4324-8bcb-b80686cdfbb3"). InnerVolumeSpecName "kube-api-access-frnkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:51.592376 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.592344    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-xks-ca" pod "31b1efe6-5e7d-4324-8bcb-b80686cdfbb3" (UID: "31b1efe6-5e7d-4324-8bcb-b80686cdfbb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:31:51.592844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.592820    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1ae3e8-21e1-4283-8917-0600eafdddf5-kube-api-access-wvzw5" pod "db1ae3e8-21e1-4283-8917-0600eafdddf5" (UID: "db1ae3e8-21e1-4283-8917-0600eafdddf5"). InnerVolumeSpecName "kube-api-access-wvzw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:51.593483 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.593458    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-xks-ca" pod "db1ae3e8-21e1-4283-8917-0600eafdddf5" (UID: "db1ae3e8-21e1-4283-8917-0600eafdddf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:31:51.688771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.688736    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:51.688771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.688766    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:51.688771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.688777    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1ae3e8-21e1-4283-8917-0600eafdddf5-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:51.689143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.688788    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frnkp\" (UniqueName: \"kubernetes.io/projected/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3-kube-api-access-frnkp\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:51.689143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:51.688799    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvzw5\" (UniqueName: \"kubernetes.io/projected/db1ae3e8-21e1-4283-8917-0600eafdddf5-kube-api-access-wvzw5\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:31:52.296774 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:52.296741    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"
Nov 02 15:31:52.296984 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:52.296771    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6" event={"ID":"db1ae3e8-21e1-4283-8917-0600eafdddf5","Type":"ContainerDied","Data":"367dbd0effe94c6f0cd7bc660a7569446f50d03b386e431583a76d7e82a62f36"}
Nov 02 15:31:52.296984 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:52.296860    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="367dbd0effe94c6f0cd7bc660a7569446f50d03b386e431583a76d7e82a62f36"
Nov 02 15:31:52.298471 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:52.298451    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"
Nov 02 15:31:52.298471 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:52.298471    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx" event={"ID":"31b1efe6-5e7d-4324-8bcb-b80686cdfbb3","Type":"ContainerDied","Data":"bad3fa3e7922d0c287e76db5e40143d342abffd6e90c130c459d97d141658735"}
Nov 02 15:31:52.298633 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:52.298523    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bad3fa3e7922d0c287e76db5e40143d342abffd6e90c130c459d97d141658735"
Nov 02 15:31:57.230023 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:57.229992    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"]
Nov 02 15:31:57.230515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:31:57.230238    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy" containerID="cri-o://4e140dc090368b4e27dac35760e43959872834121491b71e38ce9a871a86ccf9" gracePeriod=30
Nov 02 15:32:01.312982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:01.312935    4891 patch_prober.go:29] interesting pod/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:32:01.313418 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:01.313001    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:32:02.373545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.373509    4891 generic.go:356] "Generic (PLEG): container finished" podID="83865724-7220-4274-9262-f2b09ff911a1" containerID="4e140dc090368b4e27dac35760e43959872834121491b71e38ce9a871a86ccf9" exitCode=0
Nov 02 15:32:02.373931 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.373587    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" event={"ID":"83865724-7220-4274-9262-f2b09ff911a1","Type":"ContainerDied","Data":"4e140dc090368b4e27dac35760e43959872834121491b71e38ce9a871a86ccf9"}
Nov 02 15:32:02.506407 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.506296    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:32:02.609811 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.609772    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-certs\") pod \"83865724-7220-4274-9262-f2b09ff911a1\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") "
Nov 02 15:32:02.610090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.609842    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-credential-socket\") pod \"83865724-7220-4274-9262-f2b09ff911a1\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") "
Nov 02 15:32:02.610090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.609917    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-envoy\") pod \"83865724-7220-4274-9262-f2b09ff911a1\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") "
Nov 02 15:32:02.610090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.609962    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-socket\") pod \"83865724-7220-4274-9262-f2b09ff911a1\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") "
Nov 02 15:32:02.610090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.610058    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83865724-7220-4274-9262-f2b09ff911a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83865724-7220-4274-9262-f2b09ff911a1-istio-podinfo\") pod \"83865724-7220-4274-9262-f2b09ff911a1\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") "
Nov 02 15:32:02.610373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.610108    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83865724-7220-4274-9262-f2b09ff911a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83865724-7220-4274-9262-f2b09ff911a1-istiod-ca-cert\") pod \"83865724-7220-4274-9262-f2b09ff911a1\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") "
Nov 02 15:32:02.610373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.610130    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-certs" pod "83865724-7220-4274-9262-f2b09ff911a1" (UID: "83865724-7220-4274-9262-f2b09ff911a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:02.610373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.610163    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-credential-socket" pod "83865724-7220-4274-9262-f2b09ff911a1" (UID: "83865724-7220-4274-9262-f2b09ff911a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:02.610373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.610139    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-data\") pod \"83865724-7220-4274-9262-f2b09ff911a1\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") "
Nov 02 15:32:02.610373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.610221    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-istio-token\" (UniqueName: \"kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-istio-token\") pod \"83865724-7220-4274-9262-f2b09ff911a1\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") "
Nov 02 15:32:02.610373 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.610298    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-socket" pod "83865724-7220-4274-9262-f2b09ff911a1" (UID: "83865724-7220-4274-9262-f2b09ff911a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:02.610695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.610376    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-kube-api-access-6d46n\" (UniqueName: \"kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-kube-api-access-6d46n\") pod \"83865724-7220-4274-9262-f2b09ff911a1\" (UID: \"83865724-7220-4274-9262-f2b09ff911a1\") "
Nov 02 15:32:02.610695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.610474    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-data" pod "83865724-7220-4274-9262-f2b09ff911a1" (UID: "83865724-7220-4274-9262-f2b09ff911a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:02.610695 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.610575    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83865724-7220-4274-9262-f2b09ff911a1-istiod-ca-cert" pod "83865724-7220-4274-9262-f2b09ff911a1" (UID: "83865724-7220-4274-9262-f2b09ff911a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:02.611538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.611516    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:02.611720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.611696    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:02.611823 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.611724    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:02.611823 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.611739    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83865724-7220-4274-9262-f2b09ff911a1-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:02.611823 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.611753    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:02.612669 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.612643    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83865724-7220-4274-9262-f2b09ff911a1-istio-podinfo" pod "83865724-7220-4274-9262-f2b09ff911a1" (UID: "83865724-7220-4274-9262-f2b09ff911a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:32:02.612736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.612647    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-istio-token" pod "83865724-7220-4274-9262-f2b09ff911a1" (UID: "83865724-7220-4274-9262-f2b09ff911a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:02.612736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.612710    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-envoy" pod "83865724-7220-4274-9262-f2b09ff911a1" (UID: "83865724-7220-4274-9262-f2b09ff911a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:02.612813 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.612774    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-kube-api-access-6d46n" pod "83865724-7220-4274-9262-f2b09ff911a1" (UID: "83865724-7220-4274-9262-f2b09ff911a1"). InnerVolumeSpecName "kube-api-access-6d46n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:02.713152 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.713119    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83865724-7220-4274-9262-f2b09ff911a1-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:02.713152 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.713149    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83865724-7220-4274-9262-f2b09ff911a1-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:02.713152 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.713159    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:02.713454 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:02.713168    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d46n\" (UniqueName: \"kubernetes.io/projected/83865724-7220-4274-9262-f2b09ff911a1-kube-api-access-6d46n\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:03.383705 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:03.383665    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"
Nov 02 15:32:03.384204 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:03.383700    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf" event={"ID":"83865724-7220-4274-9262-f2b09ff911a1","Type":"ContainerDied","Data":"e96f92e2e9aa488bd9f22a37f4eb8695d988afb9bb498e9ebf85a98c7b79f0a8"}
Nov 02 15:32:03.384204 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:03.383761    4891 scope.go:122] "RemoveContainer" containerID="4e140dc090368b4e27dac35760e43959872834121491b71e38ce9a871a86ccf9"
Nov 02 15:32:03.600322 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:03.600277    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"]
Nov 02 15:32:03.650766 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:03.650675    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba49d0-openshift-default-6b978b7557-2qxwf"]
Nov 02 15:32:04.522744 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:04.522715    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83865724-7220-4274-9262-f2b09ff911a1" path="/var/lib/kubelet/pods/83865724-7220-4274-9262-f2b09ff911a1/volumes"
Nov 02 15:32:06.426452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.426421    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"]
Nov 02 15:32:06.426907 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.426894    4891 cpu_manager.go:414] "containerMap: removing container" podUID="db1ae3e8-21e1-4283-8917-0600eafdddf5" containerName="revoke-keys"
Nov 02 15:32:06.426907 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.426908    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1ae3e8-21e1-4283-8917-0600eafdddf5" containerName="revoke-keys"
Nov 02 15:32:06.426992 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.426936    4891 cpu_manager.go:414] "containerMap: removing container" podUID="31b1efe6-5e7d-4324-8bcb-b80686cdfbb3" containerName="revoke-keys"
Nov 02 15:32:06.426992 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.426941    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b1efe6-5e7d-4324-8bcb-b80686cdfbb3" containerName="revoke-keys"
Nov 02 15:32:06.426992 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.426963    4891 cpu_manager.go:414] "containerMap: removing container" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy"
Nov 02 15:32:06.426992 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.426968    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy"
Nov 02 15:32:06.427122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.427039    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="83865724-7220-4274-9262-f2b09ff911a1" containerName="istio-proxy"
Nov 02 15:32:06.427122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.427049    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1ae3e8-21e1-4283-8917-0600eafdddf5" containerName="revoke-keys"
Nov 02 15:32:06.427122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.427060    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b1efe6-5e7d-4324-8bcb-b80686cdfbb3" containerName="revoke-keys"
Nov 02 15:32:06.431964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.431942    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:06.535102 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.535064    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:32:06.535102 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.535100    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:32:06.535421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.535160    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 15:32:06.535421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.535251    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-39098c-serving-cert\""
Nov 02 15:32:06.535421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.535279    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vkjdq\""
Nov 02 15:32:06.538917 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.538894    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"]
Nov 02 15:32:06.557160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.557131    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-metrics-certs\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:06.557370 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.557345    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a43216-2f54-4378-82c5-7081f1bec36f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:06.557498 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.557478    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cjkg\" (UniqueName: \"kubernetes.io/projected/f0a43216-2f54-4378-82c5-7081f1bec36f-kube-api-access-4cjkg\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:06.557598 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.557580    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-maas-api-tls\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:06.659158 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.659119    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cjkg\" (UniqueName: \"kubernetes.io/projected/f0a43216-2f54-4378-82c5-7081f1bec36f-kube-api-access-4cjkg\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:06.659409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.659195    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-maas-api-tls\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:06.659409 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:32:06.659285    4891 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-serving-cert: secret "maas-api-e2e-inf-a-39098c-serving-cert" not found
Nov 02 15:32:06.659409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.659353    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-metrics-certs\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:06.659409 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:32:06.659377    4891 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-maas-api-tls podName:f0a43216-2f54-4378-82c5-7081f1bec36f nodeName:}" failed. No retries permitted until 2026-11-02 15:32:07.159357932 +0000 UTC m=+2134.751974726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-maas-api-tls") pod "maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn" (UID: "f0a43216-2f54-4378-82c5-7081f1bec36f") : secret "maas-api-e2e-inf-a-39098c-serving-cert" not found
Nov 02 15:32:06.659675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.659502    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a43216-2f54-4378-82c5-7081f1bec36f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:06.660139 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.660120    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a43216-2f54-4378-82c5-7081f1bec36f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:06.661758 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.661738    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-metrics-certs\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:06.726010 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.725936    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cjkg\" (UniqueName: \"kubernetes.io/projected/f0a43216-2f54-4378-82c5-7081f1bec36f-kube-api-access-4cjkg\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:06.970486 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.970453    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"]
Nov 02 15:32:06.976411 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:06.976342    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-468af4-xj2gt"]
Nov 02 15:32:07.165673 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:07.165634    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-maas-api-tls\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:07.168165 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:07.168141    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-maas-api-tls\") pod \"maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:07.348423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:07.348390    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:07.813221 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:32:07.813191    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a43216_2f54_4378_82c5_7081f1bec36f.slice/crio-7ad87ada4b0be0760e90c967359f75313f7d144a9e9c7163b20063f1b7ec706e WatchSource:0}: Error finding container 7ad87ada4b0be0760e90c967359f75313f7d144a9e9c7163b20063f1b7ec706e: Status 404 returned error can't find the container with id 7ad87ada4b0be0760e90c967359f75313f7d144a9e9c7163b20063f1b7ec706e
Nov 02 15:32:07.823250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:07.823216    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"]
Nov 02 15:32:08.430777 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:08.430739    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn" event={"ID":"f0a43216-2f54-4378-82c5-7081f1bec36f","Type":"ContainerStarted","Data":"7ad87ada4b0be0760e90c967359f75313f7d144a9e9c7163b20063f1b7ec706e"}
Nov 02 15:32:08.523345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:08.523292    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da45e7e-015a-4afa-9e7a-ff48cc78f1a2" path="/var/lib/kubelet/pods/5da45e7e-015a-4afa-9e7a-ff48cc78f1a2/volumes"
Nov 02 15:32:11.458736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:11.458701    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn" event={"ID":"f0a43216-2f54-4378-82c5-7081f1bec36f","Type":"ContainerStarted","Data":"c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052"}
Nov 02 15:32:11.459181 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:11.458783    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:11.579070 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:11.579010    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn" podStartSLOduration=2.598970103 podStartE2EDuration="5.578993867s" totalImagesPullingTime="2.980023764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:32:11.577095321 +0000 UTC m=+2139.169712118" watchObservedRunningTime="2026-11-02 15:32:11.578993867 +0000 UTC m=+2139.171610666"
Nov 02 15:32:14.274168 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:32:14.274128    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:32:17.472600 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:17.472569    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:32:23.885896 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:23.885859    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"]
Nov 02 15:32:23.934638 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:23.934601    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-pnv52"]
Nov 02 15:32:24.522226 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:24.522195    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ebffac-d598-438e-830d-7fc9a70dc69a" path="/var/lib/kubelet/pods/66ebffac-d598-438e-830d-7fc9a70dc69a/volumes"
Nov 02 15:32:26.661449 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.661408    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"]
Nov 02 15:32:26.668208 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.668171    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:26.720294 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.720259    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"]
Nov 02 15:32:26.765565 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.765528    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:32:26.765751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.765632    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:32:26.789199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.789159    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lhv4\" (UniqueName: \"kubernetes.io/projected/0e8358e2-c7c5-473d-b568-f747b4a75321-kube-api-access-5lhv4\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n\" (UID: \"0e8358e2-c7c5-473d-b568-f747b4a75321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:26.789406 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.789228    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n\" (UID: \"0e8358e2-c7c5-473d-b568-f747b4a75321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:26.789406 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.789295    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n\" (UID: \"0e8358e2-c7c5-473d-b568-f747b4a75321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:26.890617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.890570    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n\" (UID: \"0e8358e2-c7c5-473d-b568-f747b4a75321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:26.890825 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.890624    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n\" (UID: \"0e8358e2-c7c5-473d-b568-f747b4a75321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:26.890825 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.890791    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lhv4\" (UniqueName: \"kubernetes.io/projected/0e8358e2-c7c5-473d-b568-f747b4a75321-kube-api-access-5lhv4\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n\" (UID: \"0e8358e2-c7c5-473d-b568-f747b4a75321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:26.891506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.891484    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n\" (UID: \"0e8358e2-c7c5-473d-b568-f747b4a75321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:26.893040 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.893021    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n\" (UID: \"0e8358e2-c7c5-473d-b568-f747b4a75321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:26.952198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.952125    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lhv4\" (UniqueName: \"kubernetes.io/projected/0e8358e2-c7c5-473d-b568-f747b4a75321-kube-api-access-5lhv4\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n\" (UID: \"0e8358e2-c7c5-473d-b568-f747b4a75321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:26.982292 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:26.982252    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:27.361977 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:27.361945    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"]
Nov 02 15:32:27.363406 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:32:27.363381    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8358e2_c7c5_473d_b568_f747b4a75321.slice/crio-6129b3679677eefbc2f044a01ab28759fe8c00068458bf65d6d7d84f59e45a46 WatchSource:0}: Error finding container 6129b3679677eefbc2f044a01ab28759fe8c00068458bf65d6d7d84f59e45a46: Status 404 returned error can't find the container with id 6129b3679677eefbc2f044a01ab28759fe8c00068458bf65d6d7d84f59e45a46
Nov 02 15:32:27.576008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:27.575972    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n" event={"ID":"0e8358e2-c7c5-473d-b568-f747b4a75321","Type":"ContainerStarted","Data":"6129b3679677eefbc2f044a01ab28759fe8c00068458bf65d6d7d84f59e45a46"}
Nov 02 15:32:28.584856 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:28.584819    4891 generic.go:356] "Generic (PLEG): container finished" podID="0e8358e2-c7c5-473d-b568-f747b4a75321" containerID="1318caefb90420c6948b2a7b05abf395d4bb61c80c366d57dbb61543e93acd5b" exitCode=0
Nov 02 15:32:28.585238 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:28.584908    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n" event={"ID":"0e8358e2-c7c5-473d-b568-f747b4a75321","Type":"ContainerDied","Data":"1318caefb90420c6948b2a7b05abf395d4bb61c80c366d57dbb61543e93acd5b"}
Nov 02 15:32:29.761737 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:29.761710    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:29.838119 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:29.838084    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8358e2-c7c5-473d-b568-f747b4a75321-kube-api-access-5lhv4\" (UniqueName: \"kubernetes.io/projected/0e8358e2-c7c5-473d-b568-f747b4a75321-kube-api-access-5lhv4\") pod \"0e8358e2-c7c5-473d-b568-f747b4a75321\" (UID: \"0e8358e2-c7c5-473d-b568-f747b4a75321\") "
Nov 02 15:32:29.838293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:29.838176    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-openshift-ca\") pod \"0e8358e2-c7c5-473d-b568-f747b4a75321\" (UID: \"0e8358e2-c7c5-473d-b568-f747b4a75321\") "
Nov 02 15:32:29.838293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:29.838221    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-xks-ca\") pod \"0e8358e2-c7c5-473d-b568-f747b4a75321\" (UID: \"0e8358e2-c7c5-473d-b568-f747b4a75321\") "
Nov 02 15:32:29.838616 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:29.838583    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-openshift-ca" pod "0e8358e2-c7c5-473d-b568-f747b4a75321" (UID: "0e8358e2-c7c5-473d-b568-f747b4a75321"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:29.840415 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:29.840369    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-xks-ca" pod "0e8358e2-c7c5-473d-b568-f747b4a75321" (UID: "0e8358e2-c7c5-473d-b568-f747b4a75321"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:32:29.840504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:29.840432    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8358e2-c7c5-473d-b568-f747b4a75321-kube-api-access-5lhv4" pod "0e8358e2-c7c5-473d-b568-f747b4a75321" (UID: "0e8358e2-c7c5-473d-b568-f747b4a75321"). InnerVolumeSpecName "kube-api-access-5lhv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:29.940078 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:29.940042    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:29.940078 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:29.940072    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8358e2-c7c5-473d-b568-f747b4a75321-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:29.940078 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:29.940085    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lhv4\" (UniqueName: \"kubernetes.io/projected/0e8358e2-c7c5-473d-b568-f747b4a75321-kube-api-access-5lhv4\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:30.281242 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:30.281190    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"]
Nov 02 15:32:30.285078 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:30.285048    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d6b9c6-gdbh7"]
Nov 02 15:32:30.522912 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:30.522864    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e" path="/var/lib/kubelet/pods/44b1741a-c4c0-4e0a-819f-d9c4a6e60f9e/volumes"
Nov 02 15:32:30.600347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:30.600296    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n" event={"ID":"0e8358e2-c7c5-473d-b568-f747b4a75321","Type":"ContainerDied","Data":"6129b3679677eefbc2f044a01ab28759fe8c00068458bf65d6d7d84f59e45a46"}
Nov 02 15:32:30.600347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:30.600336    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"
Nov 02 15:32:30.600550 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:30.600360    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6129b3679677eefbc2f044a01ab28759fe8c00068458bf65d6d7d84f59e45a46"
Nov 02 15:32:31.570738 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:31.570701    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"]
Nov 02 15:32:31.620037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:31.620005    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-885c69-z8mnp"]
Nov 02 15:32:32.524586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:32.524558    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4" path="/var/lib/kubelet/pods/bba2ab6e-7dd4-4b12-8d0d-4fe378c22dd4/volumes"
Nov 02 15:32:38.274741 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:38.274703    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"]
Nov 02 15:32:38.388736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:38.388700    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"]
Nov 02 15:32:43.275409 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.275362    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5" podUID="16b6843d-d0bb-474e-94c9-e3319e14b84d" containerName="payload-processing" containerID="cri-o://2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17" gracePeriod=25
Nov 02 15:32:43.389813 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.389758    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8" podUID="addd08e1-96c5-4e16-9d9f-9eb18b1a643c" containerName="payload-pre-processing" containerID="cri-o://c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83" gracePeriod=25
Nov 02 15:32:43.558284 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.558253    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"
Nov 02 15:32:43.573227 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.573195    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"]
Nov 02 15:32:43.602587 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.602554    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b6843d-d0bb-474e-94c9-e3319e14b84d-kube-api-access-kkhz5\" (UniqueName: \"kubernetes.io/projected/16b6843d-d0bb-474e-94c9-e3319e14b84d-kube-api-access-kkhz5\") pod \"16b6843d-d0bb-474e-94c9-e3319e14b84d\" (UID: \"16b6843d-d0bb-474e-94c9-e3319e14b84d\") "
Nov 02 15:32:43.602773 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.602675    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b6843d-d0bb-474e-94c9-e3319e14b84d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b6843d-d0bb-474e-94c9-e3319e14b84d-plugins-config-volume\") pod \"16b6843d-d0bb-474e-94c9-e3319e14b84d\" (UID: \"16b6843d-d0bb-474e-94c9-e3319e14b84d\") "
Nov 02 15:32:43.603115 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.603089    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b6843d-d0bb-474e-94c9-e3319e14b84d-plugins-config-volume" pod "16b6843d-d0bb-474e-94c9-e3319e14b84d" (UID: "16b6843d-d0bb-474e-94c9-e3319e14b84d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:43.603232 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.603194    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b6843d-d0bb-474e-94c9-e3319e14b84d-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:43.604950 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.604925    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b6843d-d0bb-474e-94c9-e3319e14b84d-kube-api-access-kkhz5" pod "16b6843d-d0bb-474e-94c9-e3319e14b84d" (UID: "16b6843d-d0bb-474e-94c9-e3319e14b84d"). InnerVolumeSpecName "kube-api-access-kkhz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:43.624293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.624263    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b87e90-dxh7n"]
Nov 02 15:32:43.677481 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.677452    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"
Nov 02 15:32:43.701851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.701809    4891 generic.go:356] "Generic (PLEG): container finished" podID="16b6843d-d0bb-474e-94c9-e3319e14b84d" containerID="2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17" exitCode=0
Nov 02 15:32:43.702061 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.702043    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"
Nov 02 15:32:43.702697 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.702661    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5" event={"ID":"16b6843d-d0bb-474e-94c9-e3319e14b84d","Type":"ContainerDied","Data":"2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17"}
Nov 02 15:32:43.702829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.702764    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5" event={"ID":"16b6843d-d0bb-474e-94c9-e3319e14b84d","Type":"ContainerDied","Data":"e315b33310a997e3489e75ede1ddcb80c866b442c387d8c3583399b0923d1621"}
Nov 02 15:32:43.702829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.702798    4891 scope.go:122] "RemoveContainer" containerID="2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17"
Nov 02 15:32:43.703771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.703746    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-kube-api-access-rnql6\" (UniqueName: \"kubernetes.io/projected/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-kube-api-access-rnql6\") pod \"addd08e1-96c5-4e16-9d9f-9eb18b1a643c\" (UID: \"addd08e1-96c5-4e16-9d9f-9eb18b1a643c\") "
Nov 02 15:32:43.703879 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.703818    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-plugins-config-volume\") pod \"addd08e1-96c5-4e16-9d9f-9eb18b1a643c\" (UID: \"addd08e1-96c5-4e16-9d9f-9eb18b1a643c\") "
Nov 02 15:32:43.704180 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.704164    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkhz5\" (UniqueName: \"kubernetes.io/projected/16b6843d-d0bb-474e-94c9-e3319e14b84d-kube-api-access-kkhz5\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:43.704499 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.704480    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-plugins-config-volume" pod "addd08e1-96c5-4e16-9d9f-9eb18b1a643c" (UID: "addd08e1-96c5-4e16-9d9f-9eb18b1a643c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:43.705757 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.705720    4891 generic.go:356] "Generic (PLEG): container finished" podID="addd08e1-96c5-4e16-9d9f-9eb18b1a643c" containerID="c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83" exitCode=0
Nov 02 15:32:43.705863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.705825    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8" event={"ID":"addd08e1-96c5-4e16-9d9f-9eb18b1a643c","Type":"ContainerDied","Data":"c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83"}
Nov 02 15:32:43.705909 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.705866    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8" event={"ID":"addd08e1-96c5-4e16-9d9f-9eb18b1a643c","Type":"ContainerDied","Data":"e272f96612689d7afa4d984b2030732008c236c8cf270baf75be63146e132ceb"}
Nov 02 15:32:43.705951 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.705931    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"
Nov 02 15:32:43.706786 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.706762    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-kube-api-access-rnql6" pod "addd08e1-96c5-4e16-9d9f-9eb18b1a643c" (UID: "addd08e1-96c5-4e16-9d9f-9eb18b1a643c"). InnerVolumeSpecName "kube-api-access-rnql6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:43.716322 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.716293    4891 scope.go:122] "RemoveContainer" containerID="2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17"
Nov 02 15:32:43.716756 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:32:43.716724    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17\": container with ID starting with 2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17 not found: ID does not exist" containerID="2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17"
Nov 02 15:32:43.716887 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.716861    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17"} err="failed to get container status \"2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17\": rpc error: code = NotFound desc = could not find container \"2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17\": container with ID starting with 2a9349e09a87fa1fb9bea8fec8246d9fbc27f8fd01876b14f1b82922c3808e17 not found: ID does not exist"
Nov 02 15:32:43.716944 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.716891    4891 scope.go:122] "RemoveContainer" containerID="c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83"
Nov 02 15:32:43.728079 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.728059    4891 scope.go:122] "RemoveContainer" containerID="c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83"
Nov 02 15:32:43.728457 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:32:43.728438    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83\": container with ID starting with c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83 not found: ID does not exist" containerID="c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83"
Nov 02 15:32:43.728525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.728466    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83"} err="failed to get container status \"c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83\": rpc error: code = NotFound desc = could not find container \"c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83\": container with ID starting with c563395ecb46f345172875e3f29b353d87d020097266acde3b3fc7dd7c08dc83 not found: ID does not exist"
Nov 02 15:32:43.805718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.805636    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:43.805718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.805666    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnql6\" (UniqueName: \"kubernetes.io/projected/addd08e1-96c5-4e16-9d9f-9eb18b1a643c-kube-api-access-rnql6\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:43.967361 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.967290    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"]
Nov 02 15:32:43.970828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:43.970798    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b87e90-68f96cc4c7-fx9v5"]
Nov 02 15:32:44.135425 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.135266    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"]
Nov 02 15:32:44.138755 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.138722    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b87e90-76d87d9fb7-n5vf8"]
Nov 02 15:32:44.226899 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.226865    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"]
Nov 02 15:32:44.227397 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.227381    4891 cpu_manager.go:414] "containerMap: removing container" podUID="addd08e1-96c5-4e16-9d9f-9eb18b1a643c" containerName="payload-pre-processing"
Nov 02 15:32:44.227459 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.227398    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addd08e1-96c5-4e16-9d9f-9eb18b1a643c" containerName="payload-pre-processing"
Nov 02 15:32:44.227459 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.227411    4891 cpu_manager.go:414] "containerMap: removing container" podUID="16b6843d-d0bb-474e-94c9-e3319e14b84d" containerName="payload-processing"
Nov 02 15:32:44.227459 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.227417    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b6843d-d0bb-474e-94c9-e3319e14b84d" containerName="payload-processing"
Nov 02 15:32:44.227459 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.227428    4891 cpu_manager.go:414] "containerMap: removing container" podUID="0e8358e2-c7c5-473d-b568-f747b4a75321" containerName="revoke-keys"
Nov 02 15:32:44.227459 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.227433    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8358e2-c7c5-473d-b568-f747b4a75321" containerName="revoke-keys"
Nov 02 15:32:44.227646 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.227518    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="addd08e1-96c5-4e16-9d9f-9eb18b1a643c" containerName="payload-pre-processing"
Nov 02 15:32:44.227646 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.227527    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b6843d-d0bb-474e-94c9-e3319e14b84d" containerName="payload-processing"
Nov 02 15:32:44.227646 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.227533    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8358e2-c7c5-473d-b568-f747b4a75321" containerName="revoke-keys"
Nov 02 15:32:44.232166 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.232143    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"
Nov 02 15:32:44.275259 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:32:44.275223    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:32:44.286916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.286877    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-de9c4ba4\""
Nov 02 15:32:44.287420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.287133    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-de9c4ba4-dockercfg-nfls7\""
Nov 02 15:32:44.292628 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.292505    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"]
Nov 02 15:32:44.311325 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.311278    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a64dad-115a-4f3c-bce2-992e0fbf59c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8\" (UID: \"80a64dad-115a-4f3c-bce2-992e0fbf59c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"
Nov 02 15:32:44.311531 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.311507    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbnr7\" (UniqueName: \"kubernetes.io/projected/80a64dad-115a-4f3c-bce2-992e0fbf59c8-kube-api-access-tbnr7\") pod \"payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8\" (UID: \"80a64dad-115a-4f3c-bce2-992e0fbf59c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"
Nov 02 15:32:44.399392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.399296    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"]
Nov 02 15:32:44.406209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.406187    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"
Nov 02 15:32:44.412439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.412413    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnr7\" (UniqueName: \"kubernetes.io/projected/80a64dad-115a-4f3c-bce2-992e0fbf59c8-kube-api-access-tbnr7\") pod \"payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8\" (UID: \"80a64dad-115a-4f3c-bce2-992e0fbf59c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"
Nov 02 15:32:44.412575 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.412476    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a64dad-115a-4f3c-bce2-992e0fbf59c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8\" (UID: \"80a64dad-115a-4f3c-bce2-992e0fbf59c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"
Nov 02 15:32:44.413443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.413417    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a64dad-115a-4f3c-bce2-992e0fbf59c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8\" (UID: \"80a64dad-115a-4f3c-bce2-992e0fbf59c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"
Nov 02 15:32:44.464878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.464841    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"]
Nov 02 15:32:44.473915 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.473883    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnr7\" (UniqueName: \"kubernetes.io/projected/80a64dad-115a-4f3c-bce2-992e0fbf59c8-kube-api-access-tbnr7\") pod \"payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8\" (UID: \"80a64dad-115a-4f3c-bce2-992e0fbf59c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"
Nov 02 15:32:44.513792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.513761    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmk27\" (UniqueName: \"kubernetes.io/projected/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-kube-api-access-zmk27\") pod \"payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48\" (UID: \"251d111b-8c5f-4ae4-9712-e6f2a6da7cea\") " pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"
Nov 02 15:32:44.514009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.513982    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-plugins-config-volume\") pod \"payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48\" (UID: \"251d111b-8c5f-4ae4-9712-e6f2a6da7cea\") " pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"
Nov 02 15:32:44.522135 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.522108    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8358e2-c7c5-473d-b568-f747b4a75321" path="/var/lib/kubelet/pods/0e8358e2-c7c5-473d-b568-f747b4a75321/volumes"
Nov 02 15:32:44.522605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.522590    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b6843d-d0bb-474e-94c9-e3319e14b84d" path="/var/lib/kubelet/pods/16b6843d-d0bb-474e-94c9-e3319e14b84d/volumes"
Nov 02 15:32:44.522967 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.522954    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="addd08e1-96c5-4e16-9d9f-9eb18b1a643c" path="/var/lib/kubelet/pods/addd08e1-96c5-4e16-9d9f-9eb18b1a643c/volumes"
Nov 02 15:32:44.546001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.545978    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"
Nov 02 15:32:44.616227 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.616193    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-plugins-config-volume\") pod \"payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48\" (UID: \"251d111b-8c5f-4ae4-9712-e6f2a6da7cea\") " pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"
Nov 02 15:32:44.616377 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.616320    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmk27\" (UniqueName: \"kubernetes.io/projected/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-kube-api-access-zmk27\") pod \"payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48\" (UID: \"251d111b-8c5f-4ae4-9712-e6f2a6da7cea\") " pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"
Nov 02 15:32:44.617423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.617399    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-plugins-config-volume\") pod \"payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48\" (UID: \"251d111b-8c5f-4ae4-9712-e6f2a6da7cea\") " pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"
Nov 02 15:32:44.677561 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.677480    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmk27\" (UniqueName: \"kubernetes.io/projected/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-kube-api-access-zmk27\") pod \"payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48\" (UID: \"251d111b-8c5f-4ae4-9712-e6f2a6da7cea\") " pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"
Nov 02 15:32:44.720964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.720933    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"
Nov 02 15:32:44.929892 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:44.929867    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"]
Nov 02 15:32:44.931264 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:32:44.931231    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a64dad_115a_4f3c_bce2_992e0fbf59c8.slice/crio-8ec88dc3e0b8dae42046c37ab9f7d6342ccaf6651ca3ab5c6d89826db787c03d WatchSource:0}: Error finding container 8ec88dc3e0b8dae42046c37ab9f7d6342ccaf6651ca3ab5c6d89826db787c03d: Status 404 returned error can't find the container with id 8ec88dc3e0b8dae42046c37ab9f7d6342ccaf6651ca3ab5c6d89826db787c03d
Nov 02 15:32:45.134487 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:32:45.134457    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251d111b_8c5f_4ae4_9712_e6f2a6da7cea.slice/crio-5ace62e5f1a73089ccfd9c51f2e31891fc56c44d016845adacf0be7032c06be7 WatchSource:0}: Error finding container 5ace62e5f1a73089ccfd9c51f2e31891fc56c44d016845adacf0be7032c06be7: Status 404 returned error can't find the container with id 5ace62e5f1a73089ccfd9c51f2e31891fc56c44d016845adacf0be7032c06be7
Nov 02 15:32:45.135542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:45.135516    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"]
Nov 02 15:32:45.724384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:45.724345    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48" event={"ID":"251d111b-8c5f-4ae4-9712-e6f2a6da7cea","Type":"ContainerStarted","Data":"ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa"}
Nov 02 15:32:45.724867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:45.724411    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48" event={"ID":"251d111b-8c5f-4ae4-9712-e6f2a6da7cea","Type":"ContainerStarted","Data":"5ace62e5f1a73089ccfd9c51f2e31891fc56c44d016845adacf0be7032c06be7"}
Nov 02 15:32:45.724867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:45.724465    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"
Nov 02 15:32:45.726351 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:45.726300    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8" event={"ID":"80a64dad-115a-4f3c-bce2-992e0fbf59c8","Type":"ContainerStarted","Data":"28079f53c481e7b1013ffe4fbeeb70e97f20619e5610de2443a3b5da58075ab0"}
Nov 02 15:32:45.726468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:45.726371    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8" event={"ID":"80a64dad-115a-4f3c-bce2-992e0fbf59c8","Type":"ContainerStarted","Data":"8ec88dc3e0b8dae42046c37ab9f7d6342ccaf6651ca3ab5c6d89826db787c03d"}
Nov 02 15:32:45.726468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:45.726394    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"
Nov 02 15:32:45.842083 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:45.842033    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48" podStartSLOduration=1.8420185070000001 podStartE2EDuration="1.842018507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:32:45.839655081 +0000 UTC m=+2173.432271900" watchObservedRunningTime="2026-11-02 15:32:45.842018507 +0000 UTC m=+2173.434635306"
Nov 02 15:32:45.964882 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:45.964824    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8" podStartSLOduration=1.964810538 podStartE2EDuration="1.964810538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:32:45.960538719 +0000 UTC m=+2173.553155512" watchObservedRunningTime="2026-11-02 15:32:45.964810538 +0000 UTC m=+2173.557427331"
Nov 02 15:32:48.915822 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:48.915784    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"]
Nov 02 15:32:48.918606 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:48.918589    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:48.964622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:48.964583    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blfvl\" (UniqueName: \"kubernetes.io/projected/ab649d1a-82d5-499c-be05-2e0ce9d8c968-kube-api-access-blfvl\") pod \"maas-api-revoke-sub-09fdc8d7-8nmcr\" (UID: \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:48.964805 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:48.964651    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09fdc8d7-8nmcr\" (UID: \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:48.964871 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:48.964848    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09fdc8d7-8nmcr\" (UID: \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:48.973482 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:48.973447    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:32:49.023158 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.023114    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:32:49.026783 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.026737    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"]
Nov 02 15:32:49.066457 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.066425    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09fdc8d7-8nmcr\" (UID: \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:49.066625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.066468    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blfvl\" (UniqueName: \"kubernetes.io/projected/ab649d1a-82d5-499c-be05-2e0ce9d8c968-kube-api-access-blfvl\") pod \"maas-api-revoke-sub-09fdc8d7-8nmcr\" (UID: \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:49.066625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.066515    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09fdc8d7-8nmcr\" (UID: \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:49.067122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.067096    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09fdc8d7-8nmcr\" (UID: \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:49.068965 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.068939    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09fdc8d7-8nmcr\" (UID: \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:49.127593 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.127568    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blfvl\" (UniqueName: \"kubernetes.io/projected/ab649d1a-82d5-499c-be05-2e0ce9d8c968-kube-api-access-blfvl\") pod \"maas-api-revoke-sub-09fdc8d7-8nmcr\" (UID: \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:49.232765 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.232682    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:49.639138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.639105    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"]
Nov 02 15:32:49.640356 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:32:49.640302    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab649d1a_82d5_499c_be05_2e0ce9d8c968.slice/crio-dcd047346ebb6d9d3bb31fafb15b26f77ce21614e60d4d3a60e153e8d7239fa3 WatchSource:0}: Error finding container dcd047346ebb6d9d3bb31fafb15b26f77ce21614e60d4d3a60e153e8d7239fa3: Status 404 returned error can't find the container with id dcd047346ebb6d9d3bb31fafb15b26f77ce21614e60d4d3a60e153e8d7239fa3
Nov 02 15:32:49.758797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.758753    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr" event={"ID":"ab649d1a-82d5-499c-be05-2e0ce9d8c968","Type":"ContainerStarted","Data":"dcd047346ebb6d9d3bb31fafb15b26f77ce21614e60d4d3a60e153e8d7239fa3"}
Nov 02 15:32:49.881596 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.881563    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"]
Nov 02 15:32:49.888704 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:49.888637    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6dc69b-6kdnq"]
Nov 02 15:32:50.524016 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:50.523979    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf99298c-7016-4838-98eb-f729a6d2398d" path="/var/lib/kubelet/pods/cf99298c-7016-4838-98eb-f729a6d2398d/volumes"
Nov 02 15:32:50.767009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:50.766971    4891 generic.go:356] "Generic (PLEG): container finished" podID="ab649d1a-82d5-499c-be05-2e0ce9d8c968" containerID="d7be6d119ef8ff58d1f211b0f5040e213dab3637c7ec410a9ae0b8e5290c9e21" exitCode=0
Nov 02 15:32:50.767190 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:50.767058    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr" event={"ID":"ab649d1a-82d5-499c-be05-2e0ce9d8c968","Type":"ContainerDied","Data":"d7be6d119ef8ff58d1f211b0f5040e213dab3637c7ec410a9ae0b8e5290c9e21"}
Nov 02 15:32:51.923597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:51.923567    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:51.924968 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:51.924944    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"]
Nov 02 15:32:51.925461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:51.925442    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ab649d1a-82d5-499c-be05-2e0ce9d8c968" containerName="revoke-keys"
Nov 02 15:32:51.925461 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:51.925457    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab649d1a-82d5-499c-be05-2e0ce9d8c968" containerName="revoke-keys"
Nov 02 15:32:51.925644 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:51.925566    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab649d1a-82d5-499c-be05-2e0ce9d8c968" containerName="revoke-keys"
Nov 02 15:32:51.928156 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:51.928136    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:52.015365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.005157    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-xks-ca\") pod \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\" (UID: \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\") "
Nov 02 15:32:52.015365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.005416    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab649d1a-82d5-499c-be05-2e0ce9d8c968-kube-api-access-blfvl\" (UniqueName: \"kubernetes.io/projected/ab649d1a-82d5-499c-be05-2e0ce9d8c968-kube-api-access-blfvl\") pod \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\" (UID: \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\") "
Nov 02 15:32:52.015365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.005501    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-openshift-ca\") pod \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\" (UID: \"ab649d1a-82d5-499c-be05-2e0ce9d8c968\") "
Nov 02 15:32:52.015365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.005711    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k864v\" (UniqueName: \"kubernetes.io/projected/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-kube-api-access-k864v\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw\" (UID: \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:52.015365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.005794    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw\" (UID: \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:52.015365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.005853    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw\" (UID: \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:52.015365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.006065    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-openshift-ca" pod "ab649d1a-82d5-499c-be05-2e0ce9d8c968" (UID: "ab649d1a-82d5-499c-be05-2e0ce9d8c968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:52.022788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.022727    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab649d1a-82d5-499c-be05-2e0ce9d8c968-kube-api-access-blfvl" pod "ab649d1a-82d5-499c-be05-2e0ce9d8c968" (UID: "ab649d1a-82d5-499c-be05-2e0ce9d8c968"). InnerVolumeSpecName "kube-api-access-blfvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:52.023682 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.023650    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-xks-ca" pod "ab649d1a-82d5-499c-be05-2e0ce9d8c968" (UID: "ab649d1a-82d5-499c-be05-2e0ce9d8c968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:32:52.032250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.032173    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"]
Nov 02 15:32:52.107382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.107296    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k864v\" (UniqueName: \"kubernetes.io/projected/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-kube-api-access-k864v\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw\" (UID: \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:52.107564 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.107451    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw\" (UID: \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:52.107564 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.107512    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw\" (UID: \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:52.107751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.107662    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:52.107751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.107681    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab649d1a-82d5-499c-be05-2e0ce9d8c968-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:52.107751 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.107695    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blfvl\" (UniqueName: \"kubernetes.io/projected/ab649d1a-82d5-499c-be05-2e0ce9d8c968-kube-api-access-blfvl\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:52.108520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.108489    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw\" (UID: \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:52.110553 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.110523    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw\" (UID: \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:52.171660 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.171603    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k864v\" (UniqueName: \"kubernetes.io/projected/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-kube-api-access-k864v\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw\" (UID: \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:52.241107 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.241072    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:52.624321 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.624273    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"]
Nov 02 15:32:52.625605 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:32:52.625563    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab6ac8d_ee30_415b_b55c_6cd8244ff805.slice/crio-6bc761792cf40738c6b4b1a33931cd33f58abc12563a9d7b4bf3e072c5c5ae48 WatchSource:0}: Error finding container 6bc761792cf40738c6b4b1a33931cd33f58abc12563a9d7b4bf3e072c5c5ae48: Status 404 returned error can't find the container with id 6bc761792cf40738c6b4b1a33931cd33f58abc12563a9d7b4bf3e072c5c5ae48
Nov 02 15:32:52.815714 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.815672    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw" event={"ID":"7ab6ac8d-ee30-415b-b55c-6cd8244ff805","Type":"ContainerStarted","Data":"6bc761792cf40738c6b4b1a33931cd33f58abc12563a9d7b4bf3e072c5c5ae48"}
Nov 02 15:32:52.817398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.817370    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"
Nov 02 15:32:52.817559 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.817379    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr" event={"ID":"ab649d1a-82d5-499c-be05-2e0ce9d8c968","Type":"ContainerDied","Data":"dcd047346ebb6d9d3bb31fafb15b26f77ce21614e60d4d3a60e153e8d7239fa3"}
Nov 02 15:32:52.817559 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:52.817491    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcd047346ebb6d9d3bb31fafb15b26f77ce21614e60d4d3a60e153e8d7239fa3"
Nov 02 15:32:53.827176 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:53.827142    4891 generic.go:356] "Generic (PLEG): container finished" podID="7ab6ac8d-ee30-415b-b55c-6cd8244ff805" containerID="fa5e707abedea2afa1d45cc285c08b6236988fb516f25b24936bd043c397bb62" exitCode=0
Nov 02 15:32:53.827633 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:53.827191    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw" event={"ID":"7ab6ac8d-ee30-415b-b55c-6cd8244ff805","Type":"ContainerDied","Data":"fa5e707abedea2afa1d45cc285c08b6236988fb516f25b24936bd043c397bb62"}
Nov 02 15:32:55.075602 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.075574    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:55.164382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.155816    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-kube-api-access-k864v\" (UniqueName: \"kubernetes.io/projected/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-kube-api-access-k864v\") pod \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\" (UID: \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\") "
Nov 02 15:32:55.164382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.155933    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-xks-ca\") pod \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\" (UID: \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\") "
Nov 02 15:32:55.164382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.155969    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-openshift-ca\") pod \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\" (UID: \"7ab6ac8d-ee30-415b-b55c-6cd8244ff805\") "
Nov 02 15:32:55.164382 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.157244    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-openshift-ca" pod "7ab6ac8d-ee30-415b-b55c-6cd8244ff805" (UID: "7ab6ac8d-ee30-415b-b55c-6cd8244ff805"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:55.174238 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.174202    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-kube-api-access-k864v" pod "7ab6ac8d-ee30-415b-b55c-6cd8244ff805" (UID: "7ab6ac8d-ee30-415b-b55c-6cd8244ff805"). InnerVolumeSpecName "kube-api-access-k864v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:55.174502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.174381    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-xks-ca" pod "7ab6ac8d-ee30-415b-b55c-6cd8244ff805" (UID: "7ab6ac8d-ee30-415b-b55c-6cd8244ff805"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:32:55.270240 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.264965    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k864v\" (UniqueName: \"kubernetes.io/projected/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-kube-api-access-k864v\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:55.270240 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.269453    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:55.270240 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.269482    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab6ac8d-ee30-415b-b55c-6cd8244ff805-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:32:55.871223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.871180    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw" event={"ID":"7ab6ac8d-ee30-415b-b55c-6cd8244ff805","Type":"ContainerDied","Data":"6bc761792cf40738c6b4b1a33931cd33f58abc12563a9d7b4bf3e072c5c5ae48"}
Nov 02 15:32:55.871479 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.871249    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc761792cf40738c6b4b1a33931cd33f58abc12563a9d7b4bf3e072c5c5ae48"
Nov 02 15:32:55.871479 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:55.871362    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"
Nov 02 15:32:56.736826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:56.736795    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"
Nov 02 15:32:56.737247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:32:56.736884    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"
Nov 02 15:33:02.940440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:02.940406    4891 generic.go:356] "Generic (PLEG): container finished" podID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerID="195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c" exitCode=0
Nov 02 15:33:02.940956 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:02.940478    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" event={"ID":"2a444b7c-3af1-40a3-aa72-2932b17a8a54","Type":"ContainerDied","Data":"195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c"}
Nov 02 15:33:02.940956 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:02.940877    4891 scope.go:122] "RemoveContainer" containerID="195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c"
Nov 02 15:33:02.942354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:02.942292    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:33:02.942435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:02.942397    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:33:02.942480 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:02.942432    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:33:03.988356 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:03.988300    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" event={"ID":"2a444b7c-3af1-40a3-aa72-2932b17a8a54","Type":"ContainerStarted","Data":"7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e"}
Nov 02 15:33:04.043720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:04.043678    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:33:04.043942 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:04.043863    4891 patch_prober.go:29] interesting pod/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 15:33:04.043942 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:04.043907    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 15:33:05.044097 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:05.044058    4891 patch_prober.go:29] interesting pod/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 15:33:05.044635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:05.044124    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 15:33:06.043978 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:06.043946    4891 patch_prober.go:29] interesting pod/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 15:33:06.044168 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:06.043996    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 15:33:07.044145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:07.044108    4891 patch_prober.go:29] interesting pod/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 15:33:07.048508 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:07.048473    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 15:33:08.044024 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:08.043983    4891 patch_prober.go:29] interesting pod/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 15:33:08.044210 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:08.044043    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 15:33:09.113345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:09.113281    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:33:09.113868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:09.113736    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:33:09.114764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:09.114743    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:33:14.274143 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:33:14.274064    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:33:20.944358 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:20.944319    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"]
Nov 02 15:33:20.944884 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:20.944868    4891 cpu_manager.go:414] "containerMap: removing container" podUID="7ab6ac8d-ee30-415b-b55c-6cd8244ff805" containerName="revoke-keys"
Nov 02 15:33:20.944940 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:20.944887    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab6ac8d-ee30-415b-b55c-6cd8244ff805" containerName="revoke-keys"
Nov 02 15:33:20.945010 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:20.945000    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab6ac8d-ee30-415b-b55c-6cd8244ff805" containerName="revoke-keys"
Nov 02 15:33:20.949684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:20.949664    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"
Nov 02 15:33:21.002538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:21.002507    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4e4683\""
Nov 02 15:33:21.063425 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:21.063388    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"]
Nov 02 15:33:21.063863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:21.063840    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4e4683-dockercfg-8whlx\""
Nov 02 15:33:21.082998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:21.082964    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbd54\" (UniqueName: \"kubernetes.io/projected/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-kube-api-access-jbd54\") pod \"payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb\" (UID: \"24ccf1c1-6881-4b37-bf6e-883e6e26ef80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"
Nov 02 15:33:21.083284 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:21.083263    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb\" (UID: \"24ccf1c1-6881-4b37-bf6e-883e6e26ef80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"
Nov 02 15:33:21.184106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:21.184069    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb\" (UID: \"24ccf1c1-6881-4b37-bf6e-883e6e26ef80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"
Nov 02 15:33:21.184301 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:21.184144    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbd54\" (UniqueName: \"kubernetes.io/projected/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-kube-api-access-jbd54\") pod \"payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb\" (UID: \"24ccf1c1-6881-4b37-bf6e-883e6e26ef80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"
Nov 02 15:33:21.184844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:21.184818    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb\" (UID: \"24ccf1c1-6881-4b37-bf6e-883e6e26ef80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"
Nov 02 15:33:21.245429 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:21.245347    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbd54\" (UniqueName: \"kubernetes.io/projected/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-kube-api-access-jbd54\") pod \"payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb\" (UID: \"24ccf1c1-6881-4b37-bf6e-883e6e26ef80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"
Nov 02 15:33:21.264204 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:21.264168    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"
Nov 02 15:33:21.646730 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:21.646697    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"]
Nov 02 15:33:21.648042 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:33:21.648013    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ccf1c1_6881_4b37_bf6e_883e6e26ef80.slice/crio-991262d72ea1b48d00b67732e55f8577c2a380c9c87fce42c2afeae200f58820 WatchSource:0}: Error finding container 991262d72ea1b48d00b67732e55f8577c2a380c9c87fce42c2afeae200f58820: Status 404 returned error can't find the container with id 991262d72ea1b48d00b67732e55f8577c2a380c9c87fce42c2afeae200f58820
Nov 02 15:33:22.143949 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:22.143914    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb" event={"ID":"24ccf1c1-6881-4b37-bf6e-883e6e26ef80","Type":"ContainerStarted","Data":"516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414"}
Nov 02 15:33:22.144393 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:22.143973    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb" event={"ID":"24ccf1c1-6881-4b37-bf6e-883e6e26ef80","Type":"ContainerStarted","Data":"991262d72ea1b48d00b67732e55f8577c2a380c9c87fce42c2afeae200f58820"}
Nov 02 15:33:22.144393 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:22.144000    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"
Nov 02 15:33:22.312160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:22.312104    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb" podStartSLOduration=2.312090313 podStartE2EDuration="2.312090313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:33:22.307070574 +0000 UTC m=+2209.899687419" watchObservedRunningTime="2026-11-02 15:33:22.312090313 +0000 UTC m=+2209.904707120"
Nov 02 15:33:33.154044 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:33.154012    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"
Nov 02 15:33:44.273826 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:33:44.273788    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:33:46.960591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:46.960545    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"]
Nov 02 15:33:47.010020 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:47.009982    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4c5b4e-ktqdw"]
Nov 02 15:33:48.522386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:48.522355    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab6ac8d-ee30-415b-b55c-6cd8244ff805" path="/var/lib/kubelet/pods/7ab6ac8d-ee30-415b-b55c-6cd8244ff805/volumes"
Nov 02 15:33:54.959903 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:54.959857    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"]
Nov 02 15:33:54.965136 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:54.965115    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:55.018784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.018714    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:33:55.019059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.018824    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:33:55.023493 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.023467    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"]
Nov 02 15:33:55.070222 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.070175    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw\" (UID: \"a39a42c6-00da-437e-a9e2-b817223b1168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:55.070441 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.070244    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlfvb\" (UniqueName: \"kubernetes.io/projected/a39a42c6-00da-437e-a9e2-b817223b1168-kube-api-access-jlfvb\") pod \"maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw\" (UID: \"a39a42c6-00da-437e-a9e2-b817223b1168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:55.070441 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.070366    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw\" (UID: \"a39a42c6-00da-437e-a9e2-b817223b1168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:55.171637 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.171597    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlfvb\" (UniqueName: \"kubernetes.io/projected/a39a42c6-00da-437e-a9e2-b817223b1168-kube-api-access-jlfvb\") pod \"maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw\" (UID: \"a39a42c6-00da-437e-a9e2-b817223b1168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:55.171852 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.171666    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw\" (UID: \"a39a42c6-00da-437e-a9e2-b817223b1168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:55.171928 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.171845    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw\" (UID: \"a39a42c6-00da-437e-a9e2-b817223b1168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:55.172421 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.172393    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw\" (UID: \"a39a42c6-00da-437e-a9e2-b817223b1168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:55.174188 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.174170    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw\" (UID: \"a39a42c6-00da-437e-a9e2-b817223b1168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:55.233105 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.233030    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlfvb\" (UniqueName: \"kubernetes.io/projected/a39a42c6-00da-437e-a9e2-b817223b1168-kube-api-access-jlfvb\") pod \"maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw\" (UID: \"a39a42c6-00da-437e-a9e2-b817223b1168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:55.279135 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.279099    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:55.658194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:55.658163    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"]
Nov 02 15:33:55.659411 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:33:55.659384    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39a42c6_00da_437e_a9e2_b817223b1168.slice/crio-83d6cf6e4fc135d07eec6ac35e18fa29132fb0f90c04407d71b4f4ba368933be WatchSource:0}: Error finding container 83d6cf6e4fc135d07eec6ac35e18fa29132fb0f90c04407d71b4f4ba368933be: Status 404 returned error can't find the container with id 83d6cf6e4fc135d07eec6ac35e18fa29132fb0f90c04407d71b4f4ba368933be
Nov 02 15:33:56.387375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:56.387331    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw" event={"ID":"a39a42c6-00da-437e-a9e2-b817223b1168","Type":"ContainerStarted","Data":"c461f8ebe044cf57a22d77ef624e1d2f4995cf66fffe14dc13833a059bb11908"}
Nov 02 15:33:56.387839 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:56.387403    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw" event={"ID":"a39a42c6-00da-437e-a9e2-b817223b1168","Type":"ContainerStarted","Data":"83d6cf6e4fc135d07eec6ac35e18fa29132fb0f90c04407d71b4f4ba368933be"}
Nov 02 15:33:56.505029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:56.504966    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw" podStartSLOduration=2.504953287 podStartE2EDuration="2.504953287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:33:56.50234266 +0000 UTC m=+2244.094959456" watchObservedRunningTime="2026-11-02 15:33:56.504953287 +0000 UTC m=+2244.097570099"
Nov 02 15:33:57.397392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:57.397356    4891 generic.go:356] "Generic (PLEG): container finished" podID="a39a42c6-00da-437e-a9e2-b817223b1168" containerID="c461f8ebe044cf57a22d77ef624e1d2f4995cf66fffe14dc13833a059bb11908" exitCode=0
Nov 02 15:33:57.397852 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:57.397436    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw" event={"ID":"a39a42c6-00da-437e-a9e2-b817223b1168","Type":"ContainerDied","Data":"c461f8ebe044cf57a22d77ef624e1d2f4995cf66fffe14dc13833a059bb11908"}
Nov 02 15:33:58.545853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:58.545831    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:58.610329 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:58.610283    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-xks-ca\") pod \"a39a42c6-00da-437e-a9e2-b817223b1168\" (UID: \"a39a42c6-00da-437e-a9e2-b817223b1168\") "
Nov 02 15:33:58.610515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:58.610385    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-openshift-ca\") pod \"a39a42c6-00da-437e-a9e2-b817223b1168\" (UID: \"a39a42c6-00da-437e-a9e2-b817223b1168\") "
Nov 02 15:33:58.610584 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:58.610561    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39a42c6-00da-437e-a9e2-b817223b1168-kube-api-access-jlfvb\" (UniqueName: \"kubernetes.io/projected/a39a42c6-00da-437e-a9e2-b817223b1168-kube-api-access-jlfvb\") pod \"a39a42c6-00da-437e-a9e2-b817223b1168\" (UID: \"a39a42c6-00da-437e-a9e2-b817223b1168\") "
Nov 02 15:33:58.610778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:58.610749    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-openshift-ca" pod "a39a42c6-00da-437e-a9e2-b817223b1168" (UID: "a39a42c6-00da-437e-a9e2-b817223b1168"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:33:58.611007 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:58.610989    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:33:58.612244 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:58.612212    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-xks-ca" pod "a39a42c6-00da-437e-a9e2-b817223b1168" (UID: "a39a42c6-00da-437e-a9e2-b817223b1168"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:33:58.612567 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:58.612547    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39a42c6-00da-437e-a9e2-b817223b1168-kube-api-access-jlfvb" pod "a39a42c6-00da-437e-a9e2-b817223b1168" (UID: "a39a42c6-00da-437e-a9e2-b817223b1168"). InnerVolumeSpecName "kube-api-access-jlfvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:58.712768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:58.712667    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlfvb\" (UniqueName: \"kubernetes.io/projected/a39a42c6-00da-437e-a9e2-b817223b1168-kube-api-access-jlfvb\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:33:58.712768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:58.712708    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39a42c6-00da-437e-a9e2-b817223b1168-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:33:59.412121 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:59.412091    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"
Nov 02 15:33:59.412356 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:59.412127    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw" event={"ID":"a39a42c6-00da-437e-a9e2-b817223b1168","Type":"ContainerDied","Data":"83d6cf6e4fc135d07eec6ac35e18fa29132fb0f90c04407d71b4f4ba368933be"}
Nov 02 15:33:59.412356 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:59.412177    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83d6cf6e4fc135d07eec6ac35e18fa29132fb0f90c04407d71b4f4ba368933be"
Nov 02 15:33:59.576066 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:33:59.576026    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"]
Nov 02 15:34:00.497272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.497240    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"]
Nov 02 15:34:00.497828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.497812    4891 cpu_manager.go:414] "containerMap: removing container" podUID="a39a42c6-00da-437e-a9e2-b817223b1168" containerName="revoke-keys"
Nov 02 15:34:00.497880 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.497829    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39a42c6-00da-437e-a9e2-b817223b1168" containerName="revoke-keys"
Nov 02 15:34:00.497925 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.497902    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39a42c6-00da-437e-a9e2-b817223b1168" containerName="revoke-keys"
Nov 02 15:34:00.501364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.501343    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.546114 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.546080    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"]
Nov 02 15:34:00.550033 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.550011    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"]
Nov 02 15:34:00.550163 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.550126    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.553896 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.553877    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.591965 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.591927    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6x9j8\""
Nov 02 15:34:00.596243 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.596221    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"]
Nov 02 15:34:00.603391 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.603366    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"]
Nov 02 15:34:00.613435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.613408    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"]
Nov 02 15:34:00.630937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.630904    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hmzk\" (UniqueName: \"kubernetes.io/projected/8fbc97ef-c027-417f-aabd-a423f1d2f022-kube-api-access-8hmzk\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.631062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.630941    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.631062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631006    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwdlw\" (UniqueName: \"kubernetes.io/projected/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-kube-api-access-mwdlw\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.631062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631030    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fbc97ef-c027-417f-aabd-a423f1d2f022-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.631062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631048    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.631334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631064    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.631334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631093    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.631334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631169    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.631334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631209    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc99f35f-221e-44ab-bd39-87d52079f398-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.631570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631343    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.631570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631372    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.631570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631388    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.631570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631475    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.631570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631495    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.631570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631534    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.631570 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631553    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.631964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631625    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fbc97ef-c027-417f-aabd-a423f1d2f022-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.631964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631658    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.631964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631747    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.631964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631785    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.631964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631821    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9mjh\" (UniqueName: \"kubernetes.io/projected/dc99f35f-221e-44ab-bd39-87d52079f398-kube-api-access-z9mjh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.631964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631903    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.631964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631941    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fbc97ef-c027-417f-aabd-a423f1d2f022-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.632448 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.631967    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc99f35f-221e-44ab-bd39-87d52079f398-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.632448 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.632045    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc99f35f-221e-44ab-bd39-87d52079f398-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.632448 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.632085    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.632448 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.632118    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.733625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.733574    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.733807 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.733633    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.733807 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.733676    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.733807 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.733736    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.733807 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.733761    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.733988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.733811    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.733988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.733839    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.733988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.733876    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fbc97ef-c027-417f-aabd-a423f1d2f022-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.733988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.733908    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.733988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.733952    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.733988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.733985    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.734291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734016    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9mjh\" (UniqueName: \"kubernetes.io/projected/dc99f35f-221e-44ab-bd39-87d52079f398-kube-api-access-z9mjh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.734291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734059    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.734291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734097    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fbc97ef-c027-417f-aabd-a423f1d2f022-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.734291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734135    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc99f35f-221e-44ab-bd39-87d52079f398-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.734291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734170    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc99f35f-221e-44ab-bd39-87d52079f398-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.734291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734199    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.734291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734231    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.734291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734274    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hmzk\" (UniqueName: \"kubernetes.io/projected/8fbc97ef-c027-417f-aabd-a423f1d2f022-kube-api-access-8hmzk\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.734710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734303    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.734710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734325    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.734710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734196    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.734710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734355    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwdlw\" (UniqueName: \"kubernetes.io/projected/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-kube-api-access-mwdlw\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.734710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734409    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fbc97ef-c027-417f-aabd-a423f1d2f022-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.734710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734441    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.734710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734446    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.734710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734466    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.734710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734532    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.734710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734562    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.734710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734589    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc99f35f-221e-44ab-bd39-87d52079f398-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.734710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734678    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.735365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734788    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.735365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734865    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.735365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.734968    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.735642 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.735370    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.736101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.736076    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.736884 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.736856    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.737056 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.737037    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc99f35f-221e-44ab-bd39-87d52079f398-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.737668 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.737603    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fbc97ef-c027-417f-aabd-a423f1d2f022-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.737668 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.737628    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.737990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.737971    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.738477 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.738451    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fbc97ef-c027-417f-aabd-a423f1d2f022-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.738859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.738839    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.740062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.740035    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.741017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.740554    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc99f35f-221e-44ab-bd39-87d52079f398-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.741017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.740676    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fbc97ef-c027-417f-aabd-a423f1d2f022-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.741017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.740974    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.741504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.741484    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc99f35f-221e-44ab-bd39-87d52079f398-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.796278 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.796240    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.796510 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.796487    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hmzk\" (UniqueName: \"kubernetes.io/projected/8fbc97ef-c027-417f-aabd-a423f1d2f022-kube-api-access-8hmzk\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.796601 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.796549    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwdlw\" (UniqueName: \"kubernetes.io/projected/4ecc4b7a-342a-4d11-ae0d-6731df9030aa-kube-api-access-mwdlw\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-j8c7c\" (UID: \"4ecc4b7a-342a-4d11-ae0d-6731df9030aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:00.796601 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.796581    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc99f35f-221e-44ab-bd39-87d52079f398-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.796861 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.796838    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9mjh\" (UniqueName: \"kubernetes.io/projected/dc99f35f-221e-44ab-bd39-87d52079f398-kube-api-access-z9mjh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-gv6hc\" (UID: \"dc99f35f-221e-44ab-bd39-87d52079f398\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.800575 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.800544    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fbc97ef-c027-417f-aabd-a423f1d2f022-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4lr42\" (UID: \"8fbc97ef-c027-417f-aabd-a423f1d2f022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.816062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.816036    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:00.867667 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.867628    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:00.875578 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:00.875547    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:01.252526 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.252429    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"]
Nov 02 15:34:01.291223 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:34:01.291194    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbc97ef_c027_417f_aabd_a423f1d2f022.slice/crio-8b1d0554e4e7c8a1fbcadd664d4319559247ecbd4aecc88ceb0a638506c01032 WatchSource:0}: Error finding container 8b1d0554e4e7c8a1fbcadd664d4319559247ecbd4aecc88ceb0a638506c01032: Status 404 returned error can't find the container with id 8b1d0554e4e7c8a1fbcadd664d4319559247ecbd4aecc88ceb0a638506c01032
Nov 02 15:34:01.293639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.293607    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:01.293736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.293671    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:01.293736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.293702    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:01.311689 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.306739    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"]
Nov 02 15:34:01.311992 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:34:01.311947    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecc4b7a_342a_4d11_ae0d_6731df9030aa.slice/crio-75278b577cba1bbeace2c89b316addae8a0dd658d6324e1249dac016a356091a WatchSource:0}: Error finding container 75278b577cba1bbeace2c89b316addae8a0dd658d6324e1249dac016a356091a: Status 404 returned error can't find the container with id 75278b577cba1bbeace2c89b316addae8a0dd658d6324e1249dac016a356091a
Nov 02 15:34:01.313978 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.313945    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:01.314063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.314009    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:01.314063 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.314039    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:01.358494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.358467    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"]
Nov 02 15:34:01.358733 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:34:01.358709    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc99f35f_221e_44ab_bd39_87d52079f398.slice/crio-b02745539d9ac5a67077e59a809e0d44b85d8e139e16b613be3b06191a9cf517 WatchSource:0}: Error finding container b02745539d9ac5a67077e59a809e0d44b85d8e139e16b613be3b06191a9cf517: Status 404 returned error can't find the container with id b02745539d9ac5a67077e59a809e0d44b85d8e139e16b613be3b06191a9cf517
Nov 02 15:34:01.361764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.361723    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:01.361860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.361809    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:01.361906 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.361854    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:01.431367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.430266    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42" event={"ID":"8fbc97ef-c027-417f-aabd-a423f1d2f022","Type":"ContainerStarted","Data":"1293e6c1d4143967d12dc3470cb98b20868e9cab165ef88dddc1d74cd87f907e"}
Nov 02 15:34:01.431367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.430354    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42" event={"ID":"8fbc97ef-c027-417f-aabd-a423f1d2f022","Type":"ContainerStarted","Data":"8b1d0554e4e7c8a1fbcadd664d4319559247ecbd4aecc88ceb0a638506c01032"}
Nov 02 15:34:01.434814 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.433401    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" event={"ID":"dc99f35f-221e-44ab-bd39-87d52079f398","Type":"ContainerStarted","Data":"b02745539d9ac5a67077e59a809e0d44b85d8e139e16b613be3b06191a9cf517"}
Nov 02 15:34:01.436059 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.436012    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" event={"ID":"4ecc4b7a-342a-4d11-ae0d-6731df9030aa","Type":"ContainerStarted","Data":"c1c3d7c654c5f2f4d4c889b531c90c9693301391df6d4a04c95fdf9bddba1840"}
Nov 02 15:34:01.436177 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.436074    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" event={"ID":"4ecc4b7a-342a-4d11-ae0d-6731df9030aa","Type":"ContainerStarted","Data":"75278b577cba1bbeace2c89b316addae8a0dd658d6324e1249dac016a356091a"}
Nov 02 15:34:01.558425 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.558378    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42" podStartSLOduration=1.5583623420000001 podStartE2EDuration="1.558362342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:01.556067399 +0000 UTC m=+2249.148684197" watchObservedRunningTime="2026-11-02 15:34:01.558362342 +0000 UTC m=+2249.150979145"
Nov 02 15:34:01.682447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.682371    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" podStartSLOduration=1.682352074 podStartE2EDuration="1.682352074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:01.676946936 +0000 UTC m=+2249.269563736" watchObservedRunningTime="2026-11-02 15:34:01.682352074 +0000 UTC m=+2249.274968870"
Nov 02 15:34:01.816710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.816613    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:01.875987 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:01.875952    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:02.606534 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:02.605395    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" event={"ID":"dc99f35f-221e-44ab-bd39-87d52079f398","Type":"ContainerStarted","Data":"3395e6671748046cf7004af6ed5d9d2baa7694b51ec40e61579a84a2acf118d3"}
Nov 02 15:34:02.740596 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:02.740529    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" podStartSLOduration=2.740512055 podStartE2EDuration="2.740512055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:02.739977869 +0000 UTC m=+2250.332594669" watchObservedRunningTime="2026-11-02 15:34:02.740512055 +0000 UTC m=+2250.333128876"
Nov 02 15:34:02.822214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:02.822172    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4lr42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 15:34:02.822546 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:02.822510    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42" podUID="8fbc97ef-c027-417f-aabd-a423f1d2f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 15:34:02.885398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:02.875079    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:02.885398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:02.875258    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:34:02.885398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:02.875336    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" podUID="dc99f35f-221e-44ab-bd39-87d52079f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:34:02.885398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:02.879235    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:34:02.885398 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:02.879287    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" podUID="4ecc4b7a-342a-4d11-ae0d-6731df9030aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:34:03.818792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:03.818750    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4lr42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 15:34:03.820449 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:03.820414    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42" podUID="8fbc97ef-c027-417f-aabd-a423f1d2f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 15:34:03.887332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:03.871714    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:34:03.887332 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:03.871795    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" podUID="dc99f35f-221e-44ab-bd39-87d52079f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:34:03.898361 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:03.898275    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:34:03.898577 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:03.898372    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" podUID="4ecc4b7a-342a-4d11-ae0d-6731df9030aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:34:04.441987 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.441954    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"]
Nov 02 15:34:04.456335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.453774    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.522338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.519271    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-de9c4ba4-openshift-default-dockercfg-7b79t\""
Nov 02 15:34:04.655489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.649563    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.655489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.649619    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.655489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.649647    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02289c16-b904-4655-a895-125ee71ed905-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.655489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.649703    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02289c16-b904-4655-a895-125ee71ed905-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.655489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.649733    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.655489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.649769    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.655489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.649853    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.655489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.649879    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kns\" (UniqueName: \"kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-kube-api-access-b6kns\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.655489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.649973    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.731930 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.731847    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"]
Nov 02 15:34:04.776336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.753014    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.776682 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.776644    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.776911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.776886    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.777036 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.777020    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kns\" (UniqueName: \"kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-kube-api-access-b6kns\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.777245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.777225    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.777498 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.777475    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.777635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.777615    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.777758 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.777740    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02289c16-b904-4655-a895-125ee71ed905-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.777953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.777935    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02289c16-b904-4655-a895-125ee71ed905-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.778974 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.778951    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02289c16-b904-4655-a895-125ee71ed905-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.804378 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.780384    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.807694 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.793658    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.807859 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.793907    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.807983 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.794683    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.808093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.798806    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.808181 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.804143    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02289c16-b904-4655-a895-125ee71ed905-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.818891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.817990    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4lr42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 15:34:04.818891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.818063    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42" podUID="8fbc97ef-c027-417f-aabd-a423f1d2f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 15:34:04.872335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.868759    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:34:04.872335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.868839    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" podUID="dc99f35f-221e-44ab-bd39-87d52079f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:34:04.889958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.877418    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:04.889958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.877764    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:34:04.889958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.877808    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" podUID="4ecc4b7a-342a-4d11-ae0d-6731df9030aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:34:04.889958 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:04.889022    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kns\" (UniqueName: \"kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-kube-api-access-b6kns\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:05.170632 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:05.148571    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:05.817340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:05.816755    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4lr42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 15:34:05.817340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:05.816823    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42" podUID="8fbc97ef-c027-417f-aabd-a423f1d2f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 15:34:05.872335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:05.872048    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:34:05.872335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:05.872122    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" podUID="dc99f35f-221e-44ab-bd39-87d52079f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:34:05.887364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:05.887303    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:34:05.887617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:05.887591    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" podUID="4ecc4b7a-342a-4d11-ae0d-6731df9030aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:34:05.928163 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:05.928121    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"]
Nov 02 15:34:06.023781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:06.017956    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:06.023781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:06.018052    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:06.023781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:06.018098    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:06.819463 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:06.819407    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4lr42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 15:34:06.819776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:06.819747    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42" podUID="8fbc97ef-c027-417f-aabd-a423f1d2f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 15:34:06.843538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:06.843496    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" event={"ID":"02289c16-b904-4655-a895-125ee71ed905","Type":"ContainerStarted","Data":"bcc0fc03208a7b055961762ecfce8a7ac2f9492ee8792c3e0f801ffb9f14f328"}
Nov 02 15:34:06.843854 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:06.843831    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" event={"ID":"02289c16-b904-4655-a895-125ee71ed905","Type":"ContainerStarted","Data":"033e68e3e525528147eea546c180416646f27dca4c756d5addaa5eaa083a8e56"}
Nov 02 15:34:06.881810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:06.881770    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:34:06.908336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:06.907384    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" podUID="dc99f35f-221e-44ab-bd39-87d52079f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:34:06.908336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:06.892576    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:34:06.908336 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:06.907473    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" podUID="4ecc4b7a-342a-4d11-ae0d-6731df9030aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:34:07.215415 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:07.190402    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:07.317337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:07.317280    4891 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:34:07.317639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:07.317587    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" podUID="02289c16-b904-4655-a895-125ee71ed905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:34:07.848824 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:07.848781    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4lr42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 15:34:07.849275 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:07.849246    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42" podUID="8fbc97ef-c027-417f-aabd-a423f1d2f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 15:34:07.911375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:07.911339    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:34:07.912002 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:07.911954    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" podUID="dc99f35f-221e-44ab-bd39-87d52079f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:34:07.926068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:07.926034    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:34:07.926323 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:07.926284    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" podUID="4ecc4b7a-342a-4d11-ae0d-6731df9030aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:34:07.977349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:07.977079    4891 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:34:08.091027 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:08.090972    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:34:08.168397 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:08.156783    4891 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:34:08.168397 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:08.156860    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" podUID="02289c16-b904-4655-a895-125ee71ed905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:34:08.430392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:08.428971    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:34:08.817102 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:08.817053    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4lr42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 15:34:08.817293 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:08.817129    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42" podUID="8fbc97ef-c027-417f-aabd-a423f1d2f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 15:34:08.880340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:08.876597    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:34:08.880340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:08.876658    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" podUID="dc99f35f-221e-44ab-bd39-87d52079f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:34:08.880340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:08.876936    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:34:08.880340 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:08.876968    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" podUID="4ecc4b7a-342a-4d11-ae0d-6731df9030aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:34:09.163864 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:09.163760    4891 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:34:09.164484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:09.164450    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" podUID="02289c16-b904-4655-a895-125ee71ed905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:34:09.817211 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:09.817172    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4lr42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 15:34:09.817420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:09.817241    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42" podUID="8fbc97ef-c027-417f-aabd-a423f1d2f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 15:34:09.871797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:09.871754    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:34:09.872080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:09.872050    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" podUID="dc99f35f-221e-44ab-bd39-87d52079f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 15:34:09.876247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:09.876216    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:34:09.876487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:09.876462    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" podUID="4ecc4b7a-342a-4d11-ae0d-6731df9030aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 15:34:10.152978 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:10.152870    4891 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:34:10.153231 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:10.153197    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" podUID="02289c16-b904-4655-a895-125ee71ed905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:34:11.152743 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.152702    4891 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:34:11.153387 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.153355    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" podUID="02289c16-b904-4655-a895-125ee71ed905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:34:11.822484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.822434    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4lr42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:34:11.822793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.822761    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42" podUID="8fbc97ef-c027-417f-aabd-a423f1d2f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:34:11.834610 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.834580    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:11.835846 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.835822    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:11.845789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.845755    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4lr42"
Nov 02 15:34:11.893277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.893160    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:34:11.893713 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.893678    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" podUID="4ecc4b7a-342a-4d11-ae0d-6731df9030aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:34:11.914472 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.914436    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:34:11.914671 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.914498    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" podUID="dc99f35f-221e-44ab-bd39-87d52079f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:34:11.934525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.934475    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:11.935029 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.934966    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:11.936390 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:11.936367    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c"
Nov 02 15:34:12.014367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:12.014280    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" podStartSLOduration=8.014266358 podStartE2EDuration="8.014266358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:07.031526855 +0000 UTC m=+2254.624143654" watchObservedRunningTime="2026-11-02 15:34:12.014266358 +0000 UTC m=+2259.606883168"
Nov 02 15:34:12.105628 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:12.105541    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:12.106291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:12.106237    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:12.107008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:12.106989    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc"
Nov 02 15:34:12.149728 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:12.149688    4891 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:34:12.149998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:12.149967    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" podUID="02289c16-b904-4655-a895-125ee71ed905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:34:12.709057 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:12.709025    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"]
Nov 02 15:34:12.709451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:12.709238    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerName="istio-proxy" containerID="cri-o://14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05" gracePeriod=30
Nov 02 15:34:13.150188 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:13.150142    4891 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 15:34:13.150447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:13.150209    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" podUID="02289c16-b904-4655-a895-125ee71ed905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 15:34:14.153448 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.153414    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:14.153924 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.153913    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:14.154685 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.154664    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:34:14.274534 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:34:14.274497    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:34:14.597678 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.597644    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main" containerID="cri-o://f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba" gracePeriod=45
Nov 02 15:34:14.652556 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:34:14.652513    4891 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c42e0c6_3b38_4ef8_9278_1a3b0b8432e1.slice/crio-conmon-f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c42e0c6_3b38_4ef8_9278_1a3b0b8432e1.slice/crio-f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:34:14.876083 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.876056    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:34:14.911360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.911318    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:34:14.911507 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.911386    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:34:14.917731 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.917705    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-dshm\") pod \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") "
Nov 02 15:34:14.917835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.917778    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-home\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-home\") pod \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") "
Nov 02 15:34:14.917902 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.917845    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-kube-api-access-4xpcv\" (UniqueName: \"kubernetes.io/projected/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-kube-api-access-4xpcv\") pod \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") "
Nov 02 15:34:14.917988 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.917973    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-model-cache\") pod \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") "
Nov 02 15:34:14.918050 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.918028    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-home" pod "3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" (UID: "3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:34:14.918050 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.918041    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tls-certs\") pod \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") "
Nov 02 15:34:14.918138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.918117    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tmp-dir\") pod \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\" (UID: \"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1\") "
Nov 02 15:34:14.918262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.918234    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-model-cache" pod "3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" (UID: "3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:34:14.918452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.918423    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tmp-dir" pod "3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" (UID: "3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:34:14.918632 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.918615    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:14.918697 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.918638    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:14.918697 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.918653    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:14.920186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.920158    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tls-certs" pod "3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" (UID: "3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:34:14.920492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.920473    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-kube-api-access-4xpcv" pod "3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" (UID: "3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1"). InnerVolumeSpecName "kube-api-access-4xpcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:34:14.920492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:14.920476    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-dshm" pod "3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" (UID: "3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:34:15.019586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:15.019553    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xpcv\" (UniqueName: \"kubernetes.io/projected/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-kube-api-access-4xpcv\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:15.019586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:15.019578    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:15.019586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:15.019587    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:15.025335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:15.024746    4891 generic.go:356] "Generic (PLEG): container finished" podID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerID="f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba" exitCode=0
Nov 02 15:34:15.025335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:15.024771    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"
Nov 02 15:34:15.025335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:15.024802    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" event={"ID":"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1","Type":"ContainerDied","Data":"f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba"}
Nov 02 15:34:15.025335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:15.024867    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f" event={"ID":"3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1","Type":"ContainerDied","Data":"2c407b91d77c8c9877e138de58b092f3580e7eb29c5e82b8bb6e0387f6ab594f"}
Nov 02 15:34:15.025335 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:15.024892    4891 scope.go:122] "RemoveContainer" containerID="f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba"
Nov 02 15:34:15.039192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:15.039066    4891 scope.go:122] "RemoveContainer" containerID="f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba"
Nov 02 15:34:15.039469 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:34:15.039450    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba\": container with ID starting with f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba not found: ID does not exist" containerID="f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba"
Nov 02 15:34:15.039591 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:15.039477    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba"} err="failed to get container status \"f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba\": rpc error: code = NotFound desc = could not find container \"f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba\": container with ID starting with f6594d437e7c750087e8d406210ccc29eee490f93e747a4532138fea5bbe2dba not found: ID does not exist"
Nov 02 15:34:15.192810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:15.192732    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"]
Nov 02 15:34:15.199078 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:15.199044    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-42555d/simulator-w3-42555d-kserve-846b876d89-zl28f"]
Nov 02 15:34:16.522571 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:16.522539    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" path="/var/lib/kubelet/pods/3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1/volumes"
Nov 02 15:34:17.990862 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:17.990799    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:34:18.051700 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.051666    4891 generic.go:356] "Generic (PLEG): container finished" podID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerID="14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05" exitCode=0
Nov 02 15:34:18.051964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.051714    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"
Nov 02 15:34:18.051964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.051757    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" event={"ID":"d2be07f0-3816-40ea-96f3-3eced30935e9","Type":"ContainerDied","Data":"14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05"}
Nov 02 15:34:18.051964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.051825    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl" event={"ID":"d2be07f0-3816-40ea-96f3-3eced30935e9","Type":"ContainerDied","Data":"2f53f8b3dba21e4f3648577e5c025e9d91666bfd7fbac66e985fe4b8e1ef8a55"}
Nov 02 15:34:18.051964 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.051855    4891 scope.go:122] "RemoveContainer" containerID="14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05"
Nov 02 15:34:18.054150 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.054129    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-certs\") pod \"d2be07f0-3816-40ea-96f3-3eced30935e9\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") "
Nov 02 15:34:18.054305 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.054180    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-data\") pod \"d2be07f0-3816-40ea-96f3-3eced30935e9\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") "
Nov 02 15:34:18.054305 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.054219    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-token\" (UniqueName: \"kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-token\") pod \"d2be07f0-3816-40ea-96f3-3eced30935e9\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") "
Nov 02 15:34:18.054488 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.054363    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-kube-api-access-klhrf\" (UniqueName: \"kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-kube-api-access-klhrf\") pod \"d2be07f0-3816-40ea-96f3-3eced30935e9\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") "
Nov 02 15:34:18.054488 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.054392    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-credential-socket\") pod \"d2be07f0-3816-40ea-96f3-3eced30935e9\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") "
Nov 02 15:34:18.054488 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.054432    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-certs" pod "d2be07f0-3816-40ea-96f3-3eced30935e9" (UID: "d2be07f0-3816-40ea-96f3-3eced30935e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:34:18.054661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.054507    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-data" pod "d2be07f0-3816-40ea-96f3-3eced30935e9" (UID: "d2be07f0-3816-40ea-96f3-3eced30935e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:34:18.054661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.054535    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-podinfo\") pod \"d2be07f0-3816-40ea-96f3-3eced30935e9\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") "
Nov 02 15:34:18.054661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.054576    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2be07f0-3816-40ea-96f3-3eced30935e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2be07f0-3816-40ea-96f3-3eced30935e9-istiod-ca-cert\") pod \"d2be07f0-3816-40ea-96f3-3eced30935e9\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") "
Nov 02 15:34:18.054661 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.054638    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-envoy\") pod \"d2be07f0-3816-40ea-96f3-3eced30935e9\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") "
Nov 02 15:34:18.054910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.054663    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-socket\") pod \"d2be07f0-3816-40ea-96f3-3eced30935e9\" (UID: \"d2be07f0-3816-40ea-96f3-3eced30935e9\") "
Nov 02 15:34:18.054910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.054678    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-credential-socket" pod "d2be07f0-3816-40ea-96f3-3eced30935e9" (UID: "d2be07f0-3816-40ea-96f3-3eced30935e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:34:18.055035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.055012    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-socket" pod "d2be07f0-3816-40ea-96f3-3eced30935e9" (UID: "d2be07f0-3816-40ea-96f3-3eced30935e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:34:18.055243 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.055217    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2be07f0-3816-40ea-96f3-3eced30935e9-istiod-ca-cert" pod "d2be07f0-3816-40ea-96f3-3eced30935e9" (UID: "d2be07f0-3816-40ea-96f3-3eced30935e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:34:18.055366 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.055346    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:18.055437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.055372    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:18.055437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.055388    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:18.055437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.055401    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:18.056775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.056752    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-podinfo" pod "d2be07f0-3816-40ea-96f3-3eced30935e9" (UID: "d2be07f0-3816-40ea-96f3-3eced30935e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:34:18.057076 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.057055    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-kube-api-access-klhrf" pod "d2be07f0-3816-40ea-96f3-3eced30935e9" (UID: "d2be07f0-3816-40ea-96f3-3eced30935e9"). InnerVolumeSpecName "kube-api-access-klhrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:34:18.057140 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.057085    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-token" pod "d2be07f0-3816-40ea-96f3-3eced30935e9" (UID: "d2be07f0-3816-40ea-96f3-3eced30935e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:34:18.057140 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.057113    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-envoy" pod "d2be07f0-3816-40ea-96f3-3eced30935e9" (UID: "d2be07f0-3816-40ea-96f3-3eced30935e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:34:18.082622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.082600    4891 scope.go:122] "RemoveContainer" containerID="14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05"
Nov 02 15:34:18.082934 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:34:18.082913    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05\": container with ID starting with 14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05 not found: ID does not exist" containerID="14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05"
Nov 02 15:34:18.082985 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.082944    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05"} err="failed to get container status \"14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05\": rpc error: code = NotFound desc = could not find container \"14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05\": container with ID starting with 14e169cfd3239cab7e5693086c115bf311a32712f1fa0d1a0c65f6bcdd91ac05 not found: ID does not exist"
Nov 02 15:34:18.156367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.156299    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:18.156367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.156362    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:18.156367 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.156373    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klhrf\" (UniqueName: \"kubernetes.io/projected/d2be07f0-3816-40ea-96f3-3eced30935e9-kube-api-access-klhrf\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:18.156598 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.156384    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2be07f0-3816-40ea-96f3-3eced30935e9-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:18.156598 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.156393    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2be07f0-3816-40ea-96f3-3eced30935e9-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:18.475937 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.475901    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"]
Nov 02 15:34:18.480593 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.480562    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-x75hl"]
Nov 02 15:34:18.522655 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:18.522625    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" path="/var/lib/kubelet/pods/d2be07f0-3816-40ea-96f3-3eced30935e9/volumes"
Nov 02 15:34:19.623985 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.623946    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"]
Nov 02 15:34:19.624721 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.624698    4891 cpu_manager.go:414] "containerMap: removing container" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerName="istio-proxy"
Nov 02 15:34:19.624824 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.624723    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerName="istio-proxy"
Nov 02 15:34:19.624824 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.624759    4891 cpu_manager.go:414] "containerMap: removing container" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main"
Nov 02 15:34:19.624824 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.624767    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main"
Nov 02 15:34:19.624984 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.624947    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2be07f0-3816-40ea-96f3-3eced30935e9" containerName="istio-proxy"
Nov 02 15:34:19.624984 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.624963    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c42e0c6-3b38-4ef8-9278-1a3b0b8432e1" containerName="main"
Nov 02 15:34:19.629974 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.629953    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.674657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.674624    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.674805 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.674673    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.674805 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.674759    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.674897 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.674818    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.674897 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.674836    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.674897 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.674871    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f05246b-1509-4466-b06c-a3fb269b2ed6-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.675009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.674935    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7kzr\" (UniqueName: \"kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-kube-api-access-d7kzr\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.675009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.674960    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.675090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.675011    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.754044 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.754013    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"]
Nov 02 15:34:19.779343 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.779271    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.779343 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.779338    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.779617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.779369    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f05246b-1509-4466-b06c-a3fb269b2ed6-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.779617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.779428    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kzr\" (UniqueName: \"kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-kube-api-access-d7kzr\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.779617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.779468    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.779617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.779500    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.779858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.779644    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.779858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.779703    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.779858 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.779751    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.780229 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.780172    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-data\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.780740 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.780713    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-credential-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.780982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.780957    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-socket\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.781118 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.781097    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-certs\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.781657 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.781634    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f05246b-1509-4466-b06c-a3fb269b2ed6-istiod-ca-cert\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.782709 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.782682    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-envoy\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.783064 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.783043    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-podinfo\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.858836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.858805    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-token\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.865427 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.865361    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7kzr\" (UniqueName: \"kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-kube-api-access-d7kzr\") pod \"e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") " pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:19.943673 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:19.943582    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:20.384979 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:20.384949    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"]
Nov 02 15:34:20.386213 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:34:20.386187    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f05246b_1509_4466_b06c_a3fb269b2ed6.slice/crio-2bd291989977acc32a6ee38946cff6fbbf6300303d46d6891498fceb2f96eb4f WatchSource:0}: Error finding container 2bd291989977acc32a6ee38946cff6fbbf6300303d46d6891498fceb2f96eb4f: Status 404 returned error can't find the container with id 2bd291989977acc32a6ee38946cff6fbbf6300303d46d6891498fceb2f96eb4f
Nov 02 15:34:20.389171 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:20.389127    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:20.389347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:20.389212    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:20.389347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:20.389255    4891 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:34:21.079781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:21.079744    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk" event={"ID":"3f05246b-1509-4466-b06c-a3fb269b2ed6","Type":"ContainerStarted","Data":"f1504348560842303a24ea8874cfeb564c10515cd297dc7bfcbc4f8a1262e18f"}
Nov 02 15:34:21.080304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:21.079801    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk" event={"ID":"3f05246b-1509-4466-b06c-a3fb269b2ed6","Type":"ContainerStarted","Data":"2bd291989977acc32a6ee38946cff6fbbf6300303d46d6891498fceb2f96eb4f"}
Nov 02 15:34:21.201671 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:21.201622    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk" podStartSLOduration=2.201609388 podStartE2EDuration="2.201609388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:21.201386938 +0000 UTC m=+2268.794003737" watchObservedRunningTime="2026-11-02 15:34:21.201609388 +0000 UTC m=+2268.794226187"
Nov 02 15:34:21.945342 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:21.944357    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:21.956344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:21.952413    4891 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 15:34:21.956344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:21.952473    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk" podUID="3f05246b-1509-4466-b06c-a3fb269b2ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 15:34:22.944566 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:22.944522    4891 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 15:34:22.945198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:22.945164    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk" podUID="3f05246b-1509-4466-b06c-a3fb269b2ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 15:34:23.944086 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:23.944050    4891 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 15:34:23.944282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:23.944101    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk" podUID="3f05246b-1509-4466-b06c-a3fb269b2ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 15:34:24.944089 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:24.944054    4891 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 15:34:24.944582 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:24.944109    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk" podUID="3f05246b-1509-4466-b06c-a3fb269b2ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 15:34:25.944477 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:25.944441    4891 patch_prober.go:29] interesting pod/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 15:34:25.945093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:25.945060    4891 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk" podUID="3f05246b-1509-4466-b06c-a3fb269b2ed6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 15:34:26.197109 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:26.197031    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6773-e1ed7d"]
Nov 02 15:34:26.201218 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:26.201198    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6773-e1ed7d"
Nov 02 15:34:26.261711 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:26.261680    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xkc9h\""
Nov 02 15:34:26.271262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:26.271232    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6773-e1ed7d"]
Nov 02 15:34:26.282970 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:26.282936    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8fx\" (UniqueName: \"kubernetes.io/projected/68aceb42-959e-4dcf-88ae-2e8638ae5eae-kube-api-access-hs8fx\") pod \"test-curl-6773-e1ed7d\" (UID: \"68aceb42-959e-4dcf-88ae-2e8638ae5eae\") " pod="opendatahub/test-curl-6773-e1ed7d"
Nov 02 15:34:26.384389 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:26.384349    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8fx\" (UniqueName: \"kubernetes.io/projected/68aceb42-959e-4dcf-88ae-2e8638ae5eae-kube-api-access-hs8fx\") pod \"test-curl-6773-e1ed7d\" (UID: \"68aceb42-959e-4dcf-88ae-2e8638ae5eae\") " pod="opendatahub/test-curl-6773-e1ed7d"
Nov 02 15:34:26.448871 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:26.448797    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8fx\" (UniqueName: \"kubernetes.io/projected/68aceb42-959e-4dcf-88ae-2e8638ae5eae-kube-api-access-hs8fx\") pod \"test-curl-6773-e1ed7d\" (UID: \"68aceb42-959e-4dcf-88ae-2e8638ae5eae\") " pod="opendatahub/test-curl-6773-e1ed7d"
Nov 02 15:34:26.514839 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:26.514804    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6773-e1ed7d"
Nov 02 15:34:26.913752 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:26.913713    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6773-e1ed7d"]
Nov 02 15:34:26.914288 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:34:26.914246    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68aceb42_959e_4dcf_88ae_2e8638ae5eae.slice/crio-ce3661816fc15df155d9f999e9664f4895f864d158019bd5ab6a21b2a9a0220f WatchSource:0}: Error finding container ce3661816fc15df155d9f999e9664f4895f864d158019bd5ab6a21b2a9a0220f: Status 404 returned error can't find the container with id ce3661816fc15df155d9f999e9664f4895f864d158019bd5ab6a21b2a9a0220f
Nov 02 15:34:27.129577 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:27.129518    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6773-e1ed7d" event={"ID":"68aceb42-959e-4dcf-88ae-2e8638ae5eae","Type":"ContainerStarted","Data":"44d6a54a80be4ab5cd9eb376ca6f9b7cbc7ef0847ff61fe498dcb0497fa49939"}
Nov 02 15:34:27.130093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:27.129582    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6773-e1ed7d" event={"ID":"68aceb42-959e-4dcf-88ae-2e8638ae5eae","Type":"ContainerStarted","Data":"ce3661816fc15df155d9f999e9664f4895f864d158019bd5ab6a21b2a9a0220f"}
Nov 02 15:34:27.243560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:27.243441    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6773-e1ed7d" podStartSLOduration=1.243425945 podStartE2EDuration="1.243425945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:27.242707583 +0000 UTC m=+2274.835324381" watchObservedRunningTime="2026-11-02 15:34:27.243425945 +0000 UTC m=+2274.836042743"
Nov 02 15:34:27.271872 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:27.271837    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:27.273413 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:27.273384    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:27.274026 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:27.274002    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:34:28.729864 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:28.729829    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6773-e1ed7d"]
Nov 02 15:34:28.736712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:28.736675    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6773-e1ed7d"]
Nov 02 15:34:29.145553 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.145508    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6773-e1ed7d" podUID="68aceb42-959e-4dcf-88ae-2e8638ae5eae" containerName="test-curl-6773-e1ed7d" containerID="cri-o://44d6a54a80be4ab5cd9eb376ca6f9b7cbc7ef0847ff61fe498dcb0497fa49939" gracePeriod=2
Nov 02 15:34:29.196612 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:34:29.196572    4891 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6773-e1ed7d\" is forbidden: User \"system:node:ip-10-0-132-27.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-27.us-west-1.compute.internal' and this object" podUID="68aceb42-959e-4dcf-88ae-2e8638ae5eae" pod="opendatahub/test-curl-6773-e1ed7d"
Nov 02 15:34:29.349190 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.349154    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"]
Nov 02 15:34:29.349778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.349761    4891 cpu_manager.go:414] "containerMap: removing container" podUID="68aceb42-959e-4dcf-88ae-2e8638ae5eae" containerName="test-curl-6773-e1ed7d"
Nov 02 15:34:29.349850 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.349780    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68aceb42-959e-4dcf-88ae-2e8638ae5eae" containerName="test-curl-6773-e1ed7d"
Nov 02 15:34:29.349888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.349878    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="68aceb42-959e-4dcf-88ae-2e8638ae5eae" containerName="test-curl-6773-e1ed7d"
Nov 02 15:34:29.353392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.353373    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:29.404559 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:34:29.404475    4891 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6773-e1ed7d\" is forbidden: User \"system:node:ip-10-0-132-27.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-27.us-west-1.compute.internal' and this object" podUID="68aceb42-959e-4dcf-88ae-2e8638ae5eae" pod="opendatahub/test-curl-6773-e1ed7d"
Nov 02 15:34:29.406492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.406471    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:34:29.424119 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.424089    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl\" (UID: \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:29.424263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.424143    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgcgn\" (UniqueName: \"kubernetes.io/projected/a57205cf-8b9c-4b23-bb23-a7810fb991ec-kube-api-access-tgcgn\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl\" (UID: \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:29.424374 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.424260    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl\" (UID: \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:29.454337 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.454281    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:34:29.460189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.460161    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"]
Nov 02 15:34:29.524907 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.524876    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl\" (UID: \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:29.525077 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.525016    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl\" (UID: \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:29.525077 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.525071    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgcgn\" (UniqueName: \"kubernetes.io/projected/a57205cf-8b9c-4b23-bb23-a7810fb991ec-kube-api-access-tgcgn\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl\" (UID: \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:29.525718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.525681    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl\" (UID: \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:29.527795 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.527770    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl\" (UID: \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:29.595150 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.595112    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgcgn\" (UniqueName: \"kubernetes.io/projected/a57205cf-8b9c-4b23-bb23-a7810fb991ec-kube-api-access-tgcgn\") pod \"maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl\" (UID: \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:29.666658 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:29.666583    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:30.048141 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:30.048111    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"]
Nov 02 15:34:30.049464 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:34:30.049434    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57205cf_8b9c_4b23_bb23_a7810fb991ec.slice/crio-3b221126e4da6e270cf4af0c90e9f0f8d368ade08c8ef01bd4e4a0a401cfe7be WatchSource:0}: Error finding container 3b221126e4da6e270cf4af0c90e9f0f8d368ade08c8ef01bd4e4a0a401cfe7be: Status 404 returned error can't find the container with id 3b221126e4da6e270cf4af0c90e9f0f8d368ade08c8ef01bd4e4a0a401cfe7be
Nov 02 15:34:30.153827 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:30.153788    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl" event={"ID":"a57205cf-8b9c-4b23-bb23-a7810fb991ec","Type":"ContainerStarted","Data":"3b221126e4da6e270cf4af0c90e9f0f8d368ade08c8ef01bd4e4a0a401cfe7be"}
Nov 02 15:34:31.164532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:31.164500    4891 generic.go:356] "Generic (PLEG): container finished" podID="a57205cf-8b9c-4b23-bb23-a7810fb991ec" containerID="9957cd1bcb70ba57a8e0fb71f6d269682d136184e869dfac7cc4a857e23e4971" exitCode=0
Nov 02 15:34:31.164896 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:31.164599    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl" event={"ID":"a57205cf-8b9c-4b23-bb23-a7810fb991ec","Type":"ContainerDied","Data":"9957cd1bcb70ba57a8e0fb71f6d269682d136184e869dfac7cc4a857e23e4971"}
Nov 02 15:34:31.400982 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:31.400954    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6773-e1ed7d"
Nov 02 15:34:31.552089 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:31.552061    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68aceb42-959e-4dcf-88ae-2e8638ae5eae-kube-api-access-hs8fx\" (UniqueName: \"kubernetes.io/projected/68aceb42-959e-4dcf-88ae-2e8638ae5eae-kube-api-access-hs8fx\") pod \"68aceb42-959e-4dcf-88ae-2e8638ae5eae\" (UID: \"68aceb42-959e-4dcf-88ae-2e8638ae5eae\") "
Nov 02 15:34:31.554146 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:31.554114    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68aceb42-959e-4dcf-88ae-2e8638ae5eae-kube-api-access-hs8fx" pod "68aceb42-959e-4dcf-88ae-2e8638ae5eae" (UID: "68aceb42-959e-4dcf-88ae-2e8638ae5eae"). InnerVolumeSpecName "kube-api-access-hs8fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:34:31.654081 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:31.654042    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs8fx\" (UniqueName: \"kubernetes.io/projected/68aceb42-959e-4dcf-88ae-2e8638ae5eae-kube-api-access-hs8fx\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:32.175599 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.175566    4891 generic.go:356] "Generic (PLEG): container finished" podID="68aceb42-959e-4dcf-88ae-2e8638ae5eae" containerID="44d6a54a80be4ab5cd9eb376ca6f9b7cbc7ef0847ff61fe498dcb0497fa49939" exitCode=137
Nov 02 15:34:32.176045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.175608    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6773-e1ed7d"
Nov 02 15:34:32.176045 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.175665    4891 scope.go:122] "RemoveContainer" containerID="44d6a54a80be4ab5cd9eb376ca6f9b7cbc7ef0847ff61fe498dcb0497fa49939"
Nov 02 15:34:32.187868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.187847    4891 scope.go:122] "RemoveContainer" containerID="44d6a54a80be4ab5cd9eb376ca6f9b7cbc7ef0847ff61fe498dcb0497fa49939"
Nov 02 15:34:32.188146 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:34:32.188124    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d6a54a80be4ab5cd9eb376ca6f9b7cbc7ef0847ff61fe498dcb0497fa49939\": container with ID starting with 44d6a54a80be4ab5cd9eb376ca6f9b7cbc7ef0847ff61fe498dcb0497fa49939 not found: ID does not exist" containerID="44d6a54a80be4ab5cd9eb376ca6f9b7cbc7ef0847ff61fe498dcb0497fa49939"
Nov 02 15:34:32.188244 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.188159    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d6a54a80be4ab5cd9eb376ca6f9b7cbc7ef0847ff61fe498dcb0497fa49939"} err="failed to get container status \"44d6a54a80be4ab5cd9eb376ca6f9b7cbc7ef0847ff61fe498dcb0497fa49939\": rpc error: code = NotFound desc = could not find container \"44d6a54a80be4ab5cd9eb376ca6f9b7cbc7ef0847ff61fe498dcb0497fa49939\": container with ID starting with 44d6a54a80be4ab5cd9eb376ca6f9b7cbc7ef0847ff61fe498dcb0497fa49939 not found: ID does not exist"
Nov 02 15:34:32.236644 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:34:32.236584    4891 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6773-e1ed7d\" is forbidden: User \"system:node:ip-10-0-132-27.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-27.us-west-1.compute.internal' and this object" podUID="68aceb42-959e-4dcf-88ae-2e8638ae5eae" pod="opendatahub/test-curl-6773-e1ed7d"
Nov 02 15:34:32.323217 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.323193    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:32.374418 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:34:32.374378    4891 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6773-e1ed7d\" is forbidden: User \"system:node:ip-10-0-132-27.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-27.us-west-1.compute.internal' and this object" podUID="68aceb42-959e-4dcf-88ae-2e8638ae5eae" pod="opendatahub/test-curl-6773-e1ed7d"
Nov 02 15:34:32.463487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.463401    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-xks-ca\") pod \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\" (UID: \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\") "
Nov 02 15:34:32.463487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.463464    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57205cf-8b9c-4b23-bb23-a7810fb991ec-kube-api-access-tgcgn\" (UniqueName: \"kubernetes.io/projected/a57205cf-8b9c-4b23-bb23-a7810fb991ec-kube-api-access-tgcgn\") pod \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\" (UID: \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\") "
Nov 02 15:34:32.463725 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.463574    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-openshift-ca\") pod \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\" (UID: \"a57205cf-8b9c-4b23-bb23-a7810fb991ec\") "
Nov 02 15:34:32.464001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.463971    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-openshift-ca" pod "a57205cf-8b9c-4b23-bb23-a7810fb991ec" (UID: "a57205cf-8b9c-4b23-bb23-a7810fb991ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:34:32.464146 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.464128    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:32.465867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.465841    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-xks-ca" pod "a57205cf-8b9c-4b23-bb23-a7810fb991ec" (UID: "a57205cf-8b9c-4b23-bb23-a7810fb991ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:34:32.465957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.465864    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57205cf-8b9c-4b23-bb23-a7810fb991ec-kube-api-access-tgcgn" pod "a57205cf-8b9c-4b23-bb23-a7810fb991ec" (UID: "a57205cf-8b9c-4b23-bb23-a7810fb991ec"). InnerVolumeSpecName "kube-api-access-tgcgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:34:32.525001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.524963    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68aceb42-959e-4dcf-88ae-2e8638ae5eae" path="/var/lib/kubelet/pods/68aceb42-959e-4dcf-88ae-2e8638ae5eae/volumes"
Nov 02 15:34:32.578384 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.567409    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57205cf-8b9c-4b23-bb23-a7810fb991ec-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:32.579260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:32.579229    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgcgn\" (UniqueName: \"kubernetes.io/projected/a57205cf-8b9c-4b23-bb23-a7810fb991ec-kube-api-access-tgcgn\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:32.591337 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:34:32.589500    4891 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6773-e1ed7d\" is forbidden: User \"system:node:ip-10-0-132-27.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-27.us-west-1.compute.internal' and this object" podUID="68aceb42-959e-4dcf-88ae-2e8638ae5eae" pod="opendatahub/test-curl-6773-e1ed7d"
Nov 02 15:34:33.185930 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:33.185896    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"
Nov 02 15:34:33.186364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:33.185928    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl" event={"ID":"a57205cf-8b9c-4b23-bb23-a7810fb991ec","Type":"ContainerDied","Data":"3b221126e4da6e270cf4af0c90e9f0f8d368ade08c8ef01bd4e4a0a401cfe7be"}
Nov 02 15:34:33.186364 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:33.186007    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b221126e4da6e270cf4af0c90e9f0f8d368ade08c8ef01bd4e4a0a401cfe7be"
Nov 02 15:34:40.863603 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:40.863564    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"]
Nov 02 15:34:41.057829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:41.057788    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"]
Nov 02 15:34:41.944253 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:41.944222    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"]
Nov 02 15:34:41.944786 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:41.944771    4891 cpu_manager.go:414] "containerMap: removing container" podUID="a57205cf-8b9c-4b23-bb23-a7810fb991ec" containerName="revoke-keys"
Nov 02 15:34:41.944850 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:41.944786    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57205cf-8b9c-4b23-bb23-a7810fb991ec" containerName="revoke-keys"
Nov 02 15:34:41.944914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:41.944904    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57205cf-8b9c-4b23-bb23-a7810fb991ec" containerName="revoke-keys"
Nov 02 15:34:41.949759 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:41.949739    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:41.995304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:41.995266    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw\" (UID: \"1734d55d-a7af-4691-a07f-c9c449e05dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:41.995521 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:41.995488    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb5dl\" (UniqueName: \"kubernetes.io/projected/1734d55d-a7af-4691-a07f-c9c449e05dea-kube-api-access-tb5dl\") pod \"maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw\" (UID: \"1734d55d-a7af-4691-a07f-c9c449e05dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:41.995611 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:41.995592    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw\" (UID: \"1734d55d-a7af-4691-a07f-c9c449e05dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:42.002175 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:42.002125    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:34:42.002433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:42.002414    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:34:42.006627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:42.006602    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"]
Nov 02 15:34:42.097258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:42.097217    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb5dl\" (UniqueName: \"kubernetes.io/projected/1734d55d-a7af-4691-a07f-c9c449e05dea-kube-api-access-tb5dl\") pod \"maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw\" (UID: \"1734d55d-a7af-4691-a07f-c9c449e05dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:42.097446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:42.097355    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw\" (UID: \"1734d55d-a7af-4691-a07f-c9c449e05dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:42.097446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:42.097423    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw\" (UID: \"1734d55d-a7af-4691-a07f-c9c449e05dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:42.098147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:42.098125    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw\" (UID: \"1734d55d-a7af-4691-a07f-c9c449e05dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:42.099919 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:42.099897    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw\" (UID: \"1734d55d-a7af-4691-a07f-c9c449e05dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:42.158563 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:42.158531    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb5dl\" (UniqueName: \"kubernetes.io/projected/1734d55d-a7af-4691-a07f-c9c449e05dea-kube-api-access-tb5dl\") pod \"maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw\" (UID: \"1734d55d-a7af-4691-a07f-c9c449e05dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:42.263206 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:42.263117    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:42.662375 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:34:42.662293    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1734d55d_a7af_4691_a07f_c9c449e05dea.slice/crio-eaa9c2768ca3fdc93edb2d125951995cc30d16bb646c9699f5688ec60037bca5 WatchSource:0}: Error finding container eaa9c2768ca3fdc93edb2d125951995cc30d16bb646c9699f5688ec60037bca5: Status 404 returned error can't find the container with id eaa9c2768ca3fdc93edb2d125951995cc30d16bb646c9699f5688ec60037bca5
Nov 02 15:34:42.662375 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:42.662345    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"]
Nov 02 15:34:43.261090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:43.261001    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw" event={"ID":"1734d55d-a7af-4691-a07f-c9c449e05dea","Type":"ContainerStarted","Data":"24f3bb8617fcdda1c0bfb0a80d850b2d49fe0956378950252c49f6a9d24d2808"}
Nov 02 15:34:43.261090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:43.261054    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw" event={"ID":"1734d55d-a7af-4691-a07f-c9c449e05dea","Type":"ContainerStarted","Data":"eaa9c2768ca3fdc93edb2d125951995cc30d16bb646c9699f5688ec60037bca5"}
Nov 02 15:34:43.378627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:43.378571    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw" podStartSLOduration=2.378554037 podStartE2EDuration="2.378554037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:43.377152729 +0000 UTC m=+2290.969769528" watchObservedRunningTime="2026-11-02 15:34:43.378554037 +0000 UTC m=+2290.971170850"
Nov 02 15:34:44.272531 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:44.272493    4891 generic.go:356] "Generic (PLEG): container finished" podID="1734d55d-a7af-4691-a07f-c9c449e05dea" containerID="24f3bb8617fcdda1c0bfb0a80d850b2d49fe0956378950252c49f6a9d24d2808" exitCode=0
Nov 02 15:34:44.273220 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:44.272783    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw" event={"ID":"1734d55d-a7af-4691-a07f-c9c449e05dea","Type":"ContainerDied","Data":"24f3bb8617fcdda1c0bfb0a80d850b2d49fe0956378950252c49f6a9d24d2808"}
Nov 02 15:34:44.276219 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:34:44.276188    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:34:45.428405 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:45.428378    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:45.543194 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:45.543103    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-openshift-ca\") pod \"1734d55d-a7af-4691-a07f-c9c449e05dea\" (UID: \"1734d55d-a7af-4691-a07f-c9c449e05dea\") "
Nov 02 15:34:45.543387 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:45.543255    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-xks-ca\") pod \"1734d55d-a7af-4691-a07f-c9c449e05dea\" (UID: \"1734d55d-a7af-4691-a07f-c9c449e05dea\") "
Nov 02 15:34:45.543387 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:45.543277    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1734d55d-a7af-4691-a07f-c9c449e05dea-kube-api-access-tb5dl\" (UniqueName: \"kubernetes.io/projected/1734d55d-a7af-4691-a07f-c9c449e05dea-kube-api-access-tb5dl\") pod \"1734d55d-a7af-4691-a07f-c9c449e05dea\" (UID: \"1734d55d-a7af-4691-a07f-c9c449e05dea\") "
Nov 02 15:34:45.543586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:45.543562    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-openshift-ca" pod "1734d55d-a7af-4691-a07f-c9c449e05dea" (UID: "1734d55d-a7af-4691-a07f-c9c449e05dea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:34:45.543798 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:45.543780    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:45.545361 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:45.545332    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-xks-ca" pod "1734d55d-a7af-4691-a07f-c9c449e05dea" (UID: "1734d55d-a7af-4691-a07f-c9c449e05dea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:34:45.545516 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:45.545483    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1734d55d-a7af-4691-a07f-c9c449e05dea-kube-api-access-tb5dl" pod "1734d55d-a7af-4691-a07f-c9c449e05dea" (UID: "1734d55d-a7af-4691-a07f-c9c449e05dea"). InnerVolumeSpecName "kube-api-access-tb5dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:34:45.645071 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:45.645035    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1734d55d-a7af-4691-a07f-c9c449e05dea-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:45.645071 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:45.645066    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb5dl\" (UniqueName: \"kubernetes.io/projected/1734d55d-a7af-4691-a07f-c9c449e05dea-kube-api-access-tb5dl\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:45.864441 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:45.864298    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48" podUID="251d111b-8c5f-4ae4-9712-e6f2a6da7cea" containerName="payload-processing" containerID="cri-o://ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa" gracePeriod=25
Nov 02 15:34:46.059067 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.059006    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8" podUID="80a64dad-115a-4f3c-bce2-992e0fbf59c8" containerName="payload-pre-processing" containerID="cri-o://28079f53c481e7b1013ffe4fbeeb70e97f20619e5610de2443a3b5da58075ab0" gracePeriod=25
Nov 02 15:34:46.173355 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.173325    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"
Nov 02 15:34:46.252446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.252393    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-kube-api-access-zmk27\" (UniqueName: \"kubernetes.io/projected/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-kube-api-access-zmk27\") pod \"251d111b-8c5f-4ae4-9712-e6f2a6da7cea\" (UID: \"251d111b-8c5f-4ae4-9712-e6f2a6da7cea\") "
Nov 02 15:34:46.252614 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.252532    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-plugins-config-volume\") pod \"251d111b-8c5f-4ae4-9712-e6f2a6da7cea\" (UID: \"251d111b-8c5f-4ae4-9712-e6f2a6da7cea\") "
Nov 02 15:34:46.253525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.253490    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-plugins-config-volume" pod "251d111b-8c5f-4ae4-9712-e6f2a6da7cea" (UID: "251d111b-8c5f-4ae4-9712-e6f2a6da7cea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:34:46.255776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.255739    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-kube-api-access-zmk27" pod "251d111b-8c5f-4ae4-9712-e6f2a6da7cea" (UID: "251d111b-8c5f-4ae4-9712-e6f2a6da7cea"). InnerVolumeSpecName "kube-api-access-zmk27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:34:46.289745 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.289684    4891 generic.go:356] "Generic (PLEG): container finished" podID="251d111b-8c5f-4ae4-9712-e6f2a6da7cea" containerID="ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa" exitCode=0
Nov 02 15:34:46.289953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.289745    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48" event={"ID":"251d111b-8c5f-4ae4-9712-e6f2a6da7cea","Type":"ContainerDied","Data":"ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa"}
Nov 02 15:34:46.289953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.289854    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48" event={"ID":"251d111b-8c5f-4ae4-9712-e6f2a6da7cea","Type":"ContainerDied","Data":"5ace62e5f1a73089ccfd9c51f2e31891fc56c44d016845adacf0be7032c06be7"}
Nov 02 15:34:46.289953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.289888    4891 scope.go:122] "RemoveContainer" containerID="ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa"
Nov 02 15:34:46.291180 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.290839    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"
Nov 02 15:34:46.292802 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.292776    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw" event={"ID":"1734d55d-a7af-4691-a07f-c9c449e05dea","Type":"ContainerDied","Data":"eaa9c2768ca3fdc93edb2d125951995cc30d16bb646c9699f5688ec60037bca5"}
Nov 02 15:34:46.292911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.292826    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaa9c2768ca3fdc93edb2d125951995cc30d16bb646c9699f5688ec60037bca5"
Nov 02 15:34:46.292911 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.292892    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"
Nov 02 15:34:46.299109 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.299086    4891 generic.go:356] "Generic (PLEG): container finished" podID="80a64dad-115a-4f3c-bce2-992e0fbf59c8" containerID="28079f53c481e7b1013ffe4fbeeb70e97f20619e5610de2443a3b5da58075ab0" exitCode=0
Nov 02 15:34:46.299233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.299140    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8" event={"ID":"80a64dad-115a-4f3c-bce2-992e0fbf59c8","Type":"ContainerDied","Data":"28079f53c481e7b1013ffe4fbeeb70e97f20619e5610de2443a3b5da58075ab0"}
Nov 02 15:34:46.306809 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.306783    4891 scope.go:122] "RemoveContainer" containerID="ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa"
Nov 02 15:34:46.307119 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:34:46.307099    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa\": container with ID starting with ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa not found: ID does not exist" containerID="ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa"
Nov 02 15:34:46.307201 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.307129    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa"} err="failed to get container status \"ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa\": rpc error: code = NotFound desc = could not find container \"ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa\": container with ID starting with ca88bf45f45318aceb853d4d67a51a20294d9a933c2ef7d999ccd8c9fde4c5aa not found: ID does not exist"
Nov 02 15:34:46.328515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.328488    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"
Nov 02 15:34:46.353963 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.353932    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmk27\" (UniqueName: \"kubernetes.io/projected/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-kube-api-access-zmk27\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:46.353963 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.353959    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251d111b-8c5f-4ae4-9712-e6f2a6da7cea-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:46.435357 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.435250    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"]
Nov 02 15:34:46.454897 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.454871    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a64dad-115a-4f3c-bce2-992e0fbf59c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a64dad-115a-4f3c-bce2-992e0fbf59c8-plugins-config-volume\") pod \"80a64dad-115a-4f3c-bce2-992e0fbf59c8\" (UID: \"80a64dad-115a-4f3c-bce2-992e0fbf59c8\") "
Nov 02 15:34:46.455088 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.454970    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a64dad-115a-4f3c-bce2-992e0fbf59c8-kube-api-access-tbnr7\" (UniqueName: \"kubernetes.io/projected/80a64dad-115a-4f3c-bce2-992e0fbf59c8-kube-api-access-tbnr7\") pod \"80a64dad-115a-4f3c-bce2-992e0fbf59c8\" (UID: \"80a64dad-115a-4f3c-bce2-992e0fbf59c8\") "
Nov 02 15:34:46.455718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.455691    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a64dad-115a-4f3c-bce2-992e0fbf59c8-plugins-config-volume" pod "80a64dad-115a-4f3c-bce2-992e0fbf59c8" (UID: "80a64dad-115a-4f3c-bce2-992e0fbf59c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:34:46.459148 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.459118    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a64dad-115a-4f3c-bce2-992e0fbf59c8-kube-api-access-tbnr7" pod "80a64dad-115a-4f3c-bce2-992e0fbf59c8" (UID: "80a64dad-115a-4f3c-bce2-992e0fbf59c8"). InnerVolumeSpecName "kube-api-access-tbnr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:34:46.473153 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.473129    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"]
Nov 02 15:34:46.476844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.476818    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"]
Nov 02 15:34:46.483563 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.483541    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de9c4ba4-768fdffc87-dmd48"]
Nov 02 15:34:46.523631 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.523601    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251d111b-8c5f-4ae4-9712-e6f2a6da7cea" path="/var/lib/kubelet/pods/251d111b-8c5f-4ae4-9712-e6f2a6da7cea/volumes"
Nov 02 15:34:46.556823 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.556788    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a64dad-115a-4f3c-bce2-992e0fbf59c8-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:46.556823 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:46.556817    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbnr7\" (UniqueName: \"kubernetes.io/projected/80a64dad-115a-4f3c-bce2-992e0fbf59c8-kube-api-access-tbnr7\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:47.310557 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:47.310527    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"
Nov 02 15:34:47.310557 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:47.310548    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8" event={"ID":"80a64dad-115a-4f3c-bce2-992e0fbf59c8","Type":"ContainerDied","Data":"8ec88dc3e0b8dae42046c37ab9f7d6342ccaf6651ca3ab5c6d89826db787c03d"}
Nov 02 15:34:47.310793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:47.310619    4891 scope.go:122] "RemoveContainer" containerID="28079f53c481e7b1013ffe4fbeeb70e97f20619e5610de2443a3b5da58075ab0"
Nov 02 15:34:47.487674 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:47.487623    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"]
Nov 02 15:34:47.493453 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:47.493423    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de9c4ba4-6564f5d4dc-866q8"]
Nov 02 15:34:48.523792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.523758    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a64dad-115a-4f3c-bce2-992e0fbf59c8" path="/var/lib/kubelet/pods/80a64dad-115a-4f3c-bce2-992e0fbf59c8/volumes"
Nov 02 15:34:48.611522 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.611492    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"]
Nov 02 15:34:48.612022 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.612008    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1734d55d-a7af-4691-a07f-c9c449e05dea" containerName="revoke-keys"
Nov 02 15:34:48.612064 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.612024    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1734d55d-a7af-4691-a07f-c9c449e05dea" containerName="revoke-keys"
Nov 02 15:34:48.612101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.612081    4891 cpu_manager.go:414] "containerMap: removing container" podUID="80a64dad-115a-4f3c-bce2-992e0fbf59c8" containerName="payload-pre-processing"
Nov 02 15:34:48.612101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.612090    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a64dad-115a-4f3c-bce2-992e0fbf59c8" containerName="payload-pre-processing"
Nov 02 15:34:48.612171 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.612104    4891 cpu_manager.go:414] "containerMap: removing container" podUID="251d111b-8c5f-4ae4-9712-e6f2a6da7cea" containerName="payload-processing"
Nov 02 15:34:48.612171 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.612110    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251d111b-8c5f-4ae4-9712-e6f2a6da7cea" containerName="payload-processing"
Nov 02 15:34:48.612235 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.612208    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a64dad-115a-4f3c-bce2-992e0fbf59c8" containerName="payload-pre-processing"
Nov 02 15:34:48.612235 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.612216    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="251d111b-8c5f-4ae4-9712-e6f2a6da7cea" containerName="payload-processing"
Nov 02 15:34:48.612235 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.612226    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="1734d55d-a7af-4691-a07f-c9c449e05dea" containerName="revoke-keys"
Nov 02 15:34:48.617015 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.616991    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:34:48.673381 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.673346    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"]
Nov 02 15:34:48.680636 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.680598    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-515aa7f2-m65s5\" (UID: \"4a836680-5c62-4207-861f-cf9b8875ae11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:34:48.680815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.680653    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb6kh\" (UniqueName: \"kubernetes.io/projected/4a836680-5c62-4207-861f-cf9b8875ae11-kube-api-access-kb6kh\") pod \"maas-api-revoke-sub-515aa7f2-m65s5\" (UID: \"4a836680-5c62-4207-861f-cf9b8875ae11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:34:48.680815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.680775    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-515aa7f2-m65s5\" (UID: \"4a836680-5c62-4207-861f-cf9b8875ae11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:34:48.718635 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.718608    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:34:48.718809 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.718723    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:34:48.781975 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.781893    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-515aa7f2-m65s5\" (UID: \"4a836680-5c62-4207-861f-cf9b8875ae11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:34:48.781975 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.781935    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb6kh\" (UniqueName: \"kubernetes.io/projected/4a836680-5c62-4207-861f-cf9b8875ae11-kube-api-access-kb6kh\") pod \"maas-api-revoke-sub-515aa7f2-m65s5\" (UID: \"4a836680-5c62-4207-861f-cf9b8875ae11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:34:48.781975 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.781971    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-515aa7f2-m65s5\" (UID: \"4a836680-5c62-4207-861f-cf9b8875ae11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:34:48.782703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.782682    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-515aa7f2-m65s5\" (UID: \"4a836680-5c62-4207-861f-cf9b8875ae11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:34:48.784289 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.784270    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-515aa7f2-m65s5\" (UID: \"4a836680-5c62-4207-861f-cf9b8875ae11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:34:48.843265 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.843234    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb6kh\" (UniqueName: \"kubernetes.io/projected/4a836680-5c62-4207-861f-cf9b8875ae11-kube-api-access-kb6kh\") pod \"maas-api-revoke-sub-515aa7f2-m65s5\" (UID: \"4a836680-5c62-4207-861f-cf9b8875ae11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:34:48.933369 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:48.933333    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:34:49.323017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:49.322983    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"]
Nov 02 15:34:49.324140 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:34:49.324108    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a836680_5c62_4207_861f_cf9b8875ae11.slice/crio-26eae1ba809e1bd08a7f65288cc77a25e1b92c5ac889b5e858505657281b3c32 WatchSource:0}: Error finding container 26eae1ba809e1bd08a7f65288cc77a25e1b92c5ac889b5e858505657281b3c32: Status 404 returned error can't find the container with id 26eae1ba809e1bd08a7f65288cc77a25e1b92c5ac889b5e858505657281b3c32
Nov 02 15:34:50.337730 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:50.337697    4891 generic.go:356] "Generic (PLEG): container finished" podID="4a836680-5c62-4207-861f-cf9b8875ae11" containerID="58c6a60485788006268d185afa8ebd983491e0ca142f65931d81bb3f1dbabb3c" exitCode=0
Nov 02 15:34:50.338217 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:50.337792    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5" event={"ID":"4a836680-5c62-4207-861f-cf9b8875ae11","Type":"ContainerDied","Data":"58c6a60485788006268d185afa8ebd983491e0ca142f65931d81bb3f1dbabb3c"}
Nov 02 15:34:50.338217 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:50.337850    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5" event={"ID":"4a836680-5c62-4207-861f-cf9b8875ae11","Type":"ContainerStarted","Data":"26eae1ba809e1bd08a7f65288cc77a25e1b92c5ac889b5e858505657281b3c32"}
Nov 02 15:34:51.564561 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:51.564533    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:34:51.733443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:51.733332    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-xks-ca\") pod \"4a836680-5c62-4207-861f-cf9b8875ae11\" (UID: \"4a836680-5c62-4207-861f-cf9b8875ae11\") "
Nov 02 15:34:51.733629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:51.733494    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-openshift-ca\") pod \"4a836680-5c62-4207-861f-cf9b8875ae11\" (UID: \"4a836680-5c62-4207-861f-cf9b8875ae11\") "
Nov 02 15:34:51.733629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:51.733531    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a836680-5c62-4207-861f-cf9b8875ae11-kube-api-access-kb6kh\" (UniqueName: \"kubernetes.io/projected/4a836680-5c62-4207-861f-cf9b8875ae11-kube-api-access-kb6kh\") pod \"4a836680-5c62-4207-861f-cf9b8875ae11\" (UID: \"4a836680-5c62-4207-861f-cf9b8875ae11\") "
Nov 02 15:34:51.734155 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:51.734120    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-openshift-ca" pod "4a836680-5c62-4207-861f-cf9b8875ae11" (UID: "4a836680-5c62-4207-861f-cf9b8875ae11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:34:51.736851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:51.736827    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-xks-ca" pod "4a836680-5c62-4207-861f-cf9b8875ae11" (UID: "4a836680-5c62-4207-861f-cf9b8875ae11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:34:51.737016 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:51.736877    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a836680-5c62-4207-861f-cf9b8875ae11-kube-api-access-kb6kh" pod "4a836680-5c62-4207-861f-cf9b8875ae11" (UID: "4a836680-5c62-4207-861f-cf9b8875ae11"). InnerVolumeSpecName "kube-api-access-kb6kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:34:51.834776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:51.834730    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:51.834776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:51.834765    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a836680-5c62-4207-861f-cf9b8875ae11-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:51.834776 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:51.834775    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb6kh\" (UniqueName: \"kubernetes.io/projected/4a836680-5c62-4207-861f-cf9b8875ae11-kube-api-access-kb6kh\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:34:52.358680 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:52.358632    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5" event={"ID":"4a836680-5c62-4207-861f-cf9b8875ae11","Type":"ContainerDied","Data":"26eae1ba809e1bd08a7f65288cc77a25e1b92c5ac889b5e858505657281b3c32"}
Nov 02 15:34:52.358903 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:52.358703    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26eae1ba809e1bd08a7f65288cc77a25e1b92c5ac889b5e858505657281b3c32"
Nov 02 15:34:52.358903 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:34:52.358781    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"
Nov 02 15:35:01.458283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.458215    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main" containerID="cri-o://34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744" gracePeriod=45
Nov 02 15:35:01.493581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.493534    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main" containerID="cri-o://f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd" gracePeriod=45
Nov 02 15:35:01.778921 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.778892    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:35:01.804468 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.804444    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:35:01.817672 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.817645    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-home\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-home\") pod \"91550145-6eaf-4595-83fa-868563047896\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") "
Nov 02 15:35:01.817803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.817702    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-tmp-dir\") pod \"91550145-6eaf-4595-83fa-868563047896\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") "
Nov 02 15:35:01.817803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.817781    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91550145-6eaf-4595-83fa-868563047896-tls-certs\" (UniqueName: \"kubernetes.io/secret/91550145-6eaf-4595-83fa-868563047896-tls-certs\") pod \"91550145-6eaf-4595-83fa-868563047896\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") "
Nov 02 15:35:01.817896 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.817870    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-model-cache\") pod \"91550145-6eaf-4595-83fa-868563047896\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") "
Nov 02 15:35:01.817952 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.817919    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91550145-6eaf-4595-83fa-868563047896-kube-api-access-8pjg7\" (UniqueName: \"kubernetes.io/projected/91550145-6eaf-4595-83fa-868563047896-kube-api-access-8pjg7\") pod \"91550145-6eaf-4595-83fa-868563047896\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") "
Nov 02 15:35:01.818013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.817925    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-home" pod "91550145-6eaf-4595-83fa-868563047896" (UID: "91550145-6eaf-4595-83fa-868563047896"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:01.818013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.817956    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-dshm\") pod \"91550145-6eaf-4595-83fa-868563047896\" (UID: \"91550145-6eaf-4595-83fa-868563047896\") "
Nov 02 15:35:01.818013 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.817999    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-tmp-dir" pod "91550145-6eaf-4595-83fa-868563047896" (UID: "91550145-6eaf-4595-83fa-868563047896"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:01.818204 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.818167    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-model-cache" pod "91550145-6eaf-4595-83fa-868563047896" (UID: "91550145-6eaf-4595-83fa-868563047896"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:01.818506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.818479    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:01.818506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.818505    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:01.818700 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.818518    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:01.819939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.819916    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91550145-6eaf-4595-83fa-868563047896-tls-certs" pod "91550145-6eaf-4595-83fa-868563047896" (UID: "91550145-6eaf-4595-83fa-868563047896"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:35:01.820025 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.819968    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-dshm" pod "91550145-6eaf-4595-83fa-868563047896" (UID: "91550145-6eaf-4595-83fa-868563047896"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:01.820073 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.820027    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91550145-6eaf-4595-83fa-868563047896-kube-api-access-8pjg7" pod "91550145-6eaf-4595-83fa-868563047896" (UID: "91550145-6eaf-4595-83fa-868563047896"). InnerVolumeSpecName "kube-api-access-8pjg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:01.919800 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.919757    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-dshm\") pod \"ca0edd53-e43a-42e6-884a-92620f077b34\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") "
Nov 02 15:35:01.919983 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.919823    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-tmp-dir\") pod \"ca0edd53-e43a-42e6-884a-92620f077b34\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") "
Nov 02 15:35:01.919983 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.919898    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-home\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-home\") pod \"ca0edd53-e43a-42e6-884a-92620f077b34\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") "
Nov 02 15:35:01.920125 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.920057    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0edd53-e43a-42e6-884a-92620f077b34-kube-api-access-h89n9\" (UniqueName: \"kubernetes.io/projected/ca0edd53-e43a-42e6-884a-92620f077b34-kube-api-access-h89n9\") pod \"ca0edd53-e43a-42e6-884a-92620f077b34\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") "
Nov 02 15:35:01.920197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.920104    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-tmp-dir" pod "ca0edd53-e43a-42e6-884a-92620f077b34" (UID: "ca0edd53-e43a-42e6-884a-92620f077b34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:01.920197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.920120    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca0edd53-e43a-42e6-884a-92620f077b34-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0edd53-e43a-42e6-884a-92620f077b34-tls-certs\") pod \"ca0edd53-e43a-42e6-884a-92620f077b34\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") "
Nov 02 15:35:01.920197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.920187    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-home" pod "ca0edd53-e43a-42e6-884a-92620f077b34" (UID: "ca0edd53-e43a-42e6-884a-92620f077b34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:01.920417 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.920285    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-model-cache\") pod \"ca0edd53-e43a-42e6-884a-92620f077b34\" (UID: \"ca0edd53-e43a-42e6-884a-92620f077b34\") "
Nov 02 15:35:01.920544 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.920519    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-model-cache" pod "ca0edd53-e43a-42e6-884a-92620f077b34" (UID: "ca0edd53-e43a-42e6-884a-92620f077b34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:01.920877 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.920853    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91550145-6eaf-4595-83fa-868563047896-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:01.920990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.920881    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:01.920990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.920895    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:01.920990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.920909    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pjg7\" (UniqueName: \"kubernetes.io/projected/91550145-6eaf-4595-83fa-868563047896-kube-api-access-8pjg7\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:01.920990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.920923    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91550145-6eaf-4595-83fa-868563047896-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:01.920990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.920936    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:01.922410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.922383    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-dshm" pod "ca0edd53-e43a-42e6-884a-92620f077b34" (UID: "ca0edd53-e43a-42e6-884a-92620f077b34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:01.922502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.922398    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca0edd53-e43a-42e6-884a-92620f077b34-tls-certs" pod "ca0edd53-e43a-42e6-884a-92620f077b34" (UID: "ca0edd53-e43a-42e6-884a-92620f077b34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:35:01.922829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:01.922813    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0edd53-e43a-42e6-884a-92620f077b34-kube-api-access-h89n9" pod "ca0edd53-e43a-42e6-884a-92620f077b34" (UID: "ca0edd53-e43a-42e6-884a-92620f077b34"). InnerVolumeSpecName "kube-api-access-h89n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:02.021788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.021758    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h89n9\" (UniqueName: \"kubernetes.io/projected/ca0edd53-e43a-42e6-884a-92620f077b34-kube-api-access-h89n9\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:02.021788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.021790    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0edd53-e43a-42e6-884a-92620f077b34-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:02.021998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.021800    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0edd53-e43a-42e6-884a-92620f077b34-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:02.478416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.478381    4891 generic.go:356] "Generic (PLEG): container finished" podID="91550145-6eaf-4595-83fa-868563047896" containerID="34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744" exitCode=0
Nov 02 15:35:02.478836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.478424    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"
Nov 02 15:35:02.478836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.478458    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" event={"ID":"91550145-6eaf-4595-83fa-868563047896","Type":"ContainerDied","Data":"34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744"}
Nov 02 15:35:02.478836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.478500    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p" event={"ID":"91550145-6eaf-4595-83fa-868563047896","Type":"ContainerDied","Data":"c04f6df4ff6f93457709607e267ee26361e0f6a5d6ed57623de7a3a522920f6c"}
Nov 02 15:35:02.478836 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.478524    4891 scope.go:122] "RemoveContainer" containerID="34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744"
Nov 02 15:35:02.481924 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.481902    4891 generic.go:356] "Generic (PLEG): container finished" podID="ca0edd53-e43a-42e6-884a-92620f077b34" containerID="f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd" exitCode=0
Nov 02 15:35:02.482037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.481941    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" event={"ID":"ca0edd53-e43a-42e6-884a-92620f077b34","Type":"ContainerDied","Data":"f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd"}
Nov 02 15:35:02.482037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.481970    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"
Nov 02 15:35:02.482125 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.481973    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll" event={"ID":"ca0edd53-e43a-42e6-884a-92620f077b34","Type":"ContainerDied","Data":"16d4daaef7396f583169e4e5192b1c0b696cc70984181d8765e09deda7eba5e6"}
Nov 02 15:35:02.490498 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.490478    4891 scope.go:122] "RemoveContainer" containerID="34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744"
Nov 02 15:35:02.490817 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:35:02.490791    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744\": container with ID starting with 34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744 not found: ID does not exist" containerID="34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744"
Nov 02 15:35:02.490903 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.490831    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744"} err="failed to get container status \"34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744\": rpc error: code = NotFound desc = could not find container \"34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744\": container with ID starting with 34b84835c9c50ecd63c99cf0aca1a8b43c9e5c9a7251358d17fbfc103b8ea744 not found: ID does not exist"
Nov 02 15:35:02.490903 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.490859    4891 scope.go:122] "RemoveContainer" containerID="f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd"
Nov 02 15:35:02.501769 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.501747    4891 scope.go:122] "RemoveContainer" containerID="f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd"
Nov 02 15:35:02.502075 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:35:02.502055    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd\": container with ID starting with f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd not found: ID does not exist" containerID="f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd"
Nov 02 15:35:02.502126 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.502083    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd"} err="failed to get container status \"f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd\": rpc error: code = NotFound desc = could not find container \"f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd\": container with ID starting with f46b23fffad20d0ba64d661ca55a76bae07503782cf4ab4e0506f34f595f70dd not found: ID does not exist"
Nov 02 15:35:02.645738 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.645700    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"]
Nov 02 15:35:02.694973 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.694936    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-de35cd/simulator-w2-de35cd-kserve-67cc89c4fc-zsnll"]
Nov 02 15:35:02.846074 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.846037    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"]
Nov 02 15:35:02.878007 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:02.877966    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-de35cd/premium-w2-de35cd-kserve-5bb6478896-v7b9p"]
Nov 02 15:35:04.377417 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.377385    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"]
Nov 02 15:35:04.377926 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.377913    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main"
Nov 02 15:35:04.377976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.377927    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main"
Nov 02 15:35:04.377976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.377938    4891 cpu_manager.go:414] "containerMap: removing container" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main"
Nov 02 15:35:04.377976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.377943    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main"
Nov 02 15:35:04.377976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.377968    4891 cpu_manager.go:414] "containerMap: removing container" podUID="4a836680-5c62-4207-861f-cf9b8875ae11" containerName="revoke-keys"
Nov 02 15:35:04.377976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.377973    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a836680-5c62-4207-861f-cf9b8875ae11" containerName="revoke-keys"
Nov 02 15:35:04.378184 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.378057    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" containerName="main"
Nov 02 15:35:04.378184 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.378064    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="91550145-6eaf-4595-83fa-868563047896" containerName="main"
Nov 02 15:35:04.378184 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.378075    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a836680-5c62-4207-861f-cf9b8875ae11" containerName="revoke-keys"
Nov 02 15:35:04.383000 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.382980    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:04.439255 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.439221    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"]
Nov 02 15:35:04.455005 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.454966    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68c393a9-rr58m\" (UID: \"57c068d1-4605-4d05-a8e0-a2661cd3831a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:04.455199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.455176    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsj95\" (UniqueName: \"kubernetes.io/projected/57c068d1-4605-4d05-a8e0-a2661cd3831a-kube-api-access-lsj95\") pod \"maas-api-revoke-sub-68c393a9-rr58m\" (UID: \"57c068d1-4605-4d05-a8e0-a2661cd3831a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:04.455268 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.455226    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68c393a9-rr58m\" (UID: \"57c068d1-4605-4d05-a8e0-a2661cd3831a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:04.484258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.484225    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:35:04.484411 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.484336    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:35:04.527048 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.527011    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91550145-6eaf-4595-83fa-868563047896" path="/var/lib/kubelet/pods/91550145-6eaf-4595-83fa-868563047896/volumes"
Nov 02 15:35:04.527639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.527609    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0edd53-e43a-42e6-884a-92620f077b34" path="/var/lib/kubelet/pods/ca0edd53-e43a-42e6-884a-92620f077b34/volumes"
Nov 02 15:35:04.556700 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.556662    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68c393a9-rr58m\" (UID: \"57c068d1-4605-4d05-a8e0-a2661cd3831a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:04.556914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.556771    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsj95\" (UniqueName: \"kubernetes.io/projected/57c068d1-4605-4d05-a8e0-a2661cd3831a-kube-api-access-lsj95\") pod \"maas-api-revoke-sub-68c393a9-rr58m\" (UID: \"57c068d1-4605-4d05-a8e0-a2661cd3831a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:04.556914 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.556793    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68c393a9-rr58m\" (UID: \"57c068d1-4605-4d05-a8e0-a2661cd3831a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:04.557437 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.557411    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68c393a9-rr58m\" (UID: \"57c068d1-4605-4d05-a8e0-a2661cd3831a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:04.559080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.559062    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68c393a9-rr58m\" (UID: \"57c068d1-4605-4d05-a8e0-a2661cd3831a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:04.618180 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.618145    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsj95\" (UniqueName: \"kubernetes.io/projected/57c068d1-4605-4d05-a8e0-a2661cd3831a-kube-api-access-lsj95\") pod \"maas-api-revoke-sub-68c393a9-rr58m\" (UID: \"57c068d1-4605-4d05-a8e0-a2661cd3831a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:04.697223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:04.697112    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:05.107143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:05.107110    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"]
Nov 02 15:35:05.110018 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:35:05.109990    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c068d1_4605_4d05_a8e0_a2661cd3831a.slice/crio-ab08558a9c94ddc4bede368415c550660e72869a7ebbc13dd6fb31b12a647c8e WatchSource:0}: Error finding container ab08558a9c94ddc4bede368415c550660e72869a7ebbc13dd6fb31b12a647c8e: Status 404 returned error can't find the container with id ab08558a9c94ddc4bede368415c550660e72869a7ebbc13dd6fb31b12a647c8e
Nov 02 15:35:05.548187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:05.548150    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m" event={"ID":"57c068d1-4605-4d05-a8e0-a2661cd3831a","Type":"ContainerStarted","Data":"bbe0f7131c383d63441022f05c9a14b0d3971c4a78c108b7dfca5293a1f06d04"}
Nov 02 15:35:05.548689 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:05.548210    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m" event={"ID":"57c068d1-4605-4d05-a8e0-a2661cd3831a","Type":"ContainerStarted","Data":"ab08558a9c94ddc4bede368415c550660e72869a7ebbc13dd6fb31b12a647c8e"}
Nov 02 15:35:05.669687 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:05.669575    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m" podStartSLOduration=1.669559462 podStartE2EDuration="1.669559462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:35:05.66712807 +0000 UTC m=+2313.259744871" watchObservedRunningTime="2026-11-02 15:35:05.669559462 +0000 UTC m=+2313.262176265"
Nov 02 15:35:06.558826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:06.558794    4891 generic.go:356] "Generic (PLEG): container finished" podID="57c068d1-4605-4d05-a8e0-a2661cd3831a" containerID="bbe0f7131c383d63441022f05c9a14b0d3971c4a78c108b7dfca5293a1f06d04" exitCode=0
Nov 02 15:35:06.559223 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:06.558877    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m" event={"ID":"57c068d1-4605-4d05-a8e0-a2661cd3831a","Type":"ContainerDied","Data":"bbe0f7131c383d63441022f05c9a14b0d3971c4a78c108b7dfca5293a1f06d04"}
Nov 02 15:35:06.859497 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:06.859403    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"]
Nov 02 15:35:06.874412 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:06.874373    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de9c4ba4-mrscl"]
Nov 02 15:35:07.716128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:07.716101    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:07.806459 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:07.806424    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-openshift-ca\") pod \"57c068d1-4605-4d05-a8e0-a2661cd3831a\" (UID: \"57c068d1-4605-4d05-a8e0-a2661cd3831a\") "
Nov 02 15:35:07.806636 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:07.806563    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c068d1-4605-4d05-a8e0-a2661cd3831a-kube-api-access-lsj95\" (UniqueName: \"kubernetes.io/projected/57c068d1-4605-4d05-a8e0-a2661cd3831a-kube-api-access-lsj95\") pod \"57c068d1-4605-4d05-a8e0-a2661cd3831a\" (UID: \"57c068d1-4605-4d05-a8e0-a2661cd3831a\") "
Nov 02 15:35:07.806636 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:07.806600    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-xks-ca\") pod \"57c068d1-4605-4d05-a8e0-a2661cd3831a\" (UID: \"57c068d1-4605-4d05-a8e0-a2661cd3831a\") "
Nov 02 15:35:07.806875 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:07.806852    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-openshift-ca" pod "57c068d1-4605-4d05-a8e0-a2661cd3831a" (UID: "57c068d1-4605-4d05-a8e0-a2661cd3831a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:35:07.807079 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:07.807052    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:07.808734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:07.808709    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-xks-ca" pod "57c068d1-4605-4d05-a8e0-a2661cd3831a" (UID: "57c068d1-4605-4d05-a8e0-a2661cd3831a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:35:07.808865 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:07.808754    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c068d1-4605-4d05-a8e0-a2661cd3831a-kube-api-access-lsj95" pod "57c068d1-4605-4d05-a8e0-a2661cd3831a" (UID: "57c068d1-4605-4d05-a8e0-a2661cd3831a"). InnerVolumeSpecName "kube-api-access-lsj95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:07.908670 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:07.908591    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsj95\" (UniqueName: \"kubernetes.io/projected/57c068d1-4605-4d05-a8e0-a2661cd3831a-kube-api-access-lsj95\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:07.908670 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:07.908619    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c068d1-4605-4d05-a8e0-a2661cd3831a-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:35:08.523184 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:08.523149    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a57205cf-8b9c-4b23-bb23-a7810fb991ec" path="/var/lib/kubelet/pods/a57205cf-8b9c-4b23-bb23-a7810fb991ec/volumes"
Nov 02 15:35:08.575791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:08.575753    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m" event={"ID":"57c068d1-4605-4d05-a8e0-a2661cd3831a","Type":"ContainerDied","Data":"ab08558a9c94ddc4bede368415c550660e72869a7ebbc13dd6fb31b12a647c8e"}
Nov 02 15:35:08.575791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:08.575803    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab08558a9c94ddc4bede368415c550660e72869a7ebbc13dd6fb31b12a647c8e"
Nov 02 15:35:08.576031 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:08.575763    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"
Nov 02 15:35:14.274698 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:35:14.274652    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:35:26.016660 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.016617    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"]
Nov 02 15:35:26.017131 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.017118    4891 cpu_manager.go:414] "containerMap: removing container" podUID="57c068d1-4605-4d05-a8e0-a2661cd3831a" containerName="revoke-keys"
Nov 02 15:35:26.017176 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.017131    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c068d1-4605-4d05-a8e0-a2661cd3831a" containerName="revoke-keys"
Nov 02 15:35:26.017271 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.017262    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c068d1-4605-4d05-a8e0-a2661cd3831a" containerName="revoke-keys"
Nov 02 15:35:26.021929 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.021908    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.093615 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.093581    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"]
Nov 02 15:35:26.129388 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.129295    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-39098c\"/\"default-dockercfg-46bwf\""
Nov 02 15:35:26.129388 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.129298    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-39098c\"/\"openshift-service-ca.crt\""
Nov 02 15:35:26.129388 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.129386    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-39098c\"/\"test-model-39098c-kserve-self-signed-certs\""
Nov 02 15:35:26.129793 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.129779    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-39098c\"/\"kube-root-ca.crt\""
Nov 02 15:35:26.129936 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.129915    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdvz\" (UniqueName: \"kubernetes.io/projected/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-kube-api-access-btdvz\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.130047 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.130029    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-model-cache\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.130101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.130089    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tls-certs\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.130138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.130114    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-dshm\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.130177 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.130138    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tmp-dir\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.130320 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.130295    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-home\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.231363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.231327    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tls-certs\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.231363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.231369    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-dshm\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.231623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.231388    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tmp-dir\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.231623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.231456    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-home\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.231623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.231509    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdvz\" (UniqueName: \"kubernetes.io/projected/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-kube-api-access-btdvz\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.231623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.231599    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-model-cache\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.231922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.231896    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tmp-dir\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.232023 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.231996    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-model-cache\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.232241 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.232222    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-home\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.233713 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.233692    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-dshm\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.233884 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.233867    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tls-certs\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.294154 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.294117    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdvz\" (UniqueName: \"kubernetes.io/projected/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-kube-api-access-btdvz\") pod \"test-model-39098c-kserve-5d7b4dd687-pnftg\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") " pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.334806 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.334765    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:26.736773 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:26.736745    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"]
Nov 02 15:35:26.738280 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:35:26.738236    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5419e5ef_a3cb_4a24_a8b8_fd0e9da5d7ca.slice/crio-aad96aa0bfbee42b288d07d42d1a81f8d8387af0c512779d460e785b5db536c4 WatchSource:0}: Error finding container aad96aa0bfbee42b288d07d42d1a81f8d8387af0c512779d460e785b5db536c4: Status 404 returned error can't find the container with id aad96aa0bfbee42b288d07d42d1a81f8d8387af0c512779d460e785b5db536c4
Nov 02 15:35:27.747899 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:27.747864    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg" event={"ID":"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca","Type":"ContainerStarted","Data":"58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2"}
Nov 02 15:35:27.748333 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:27.747909    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg" event={"ID":"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca","Type":"ContainerStarted","Data":"aad96aa0bfbee42b288d07d42d1a81f8d8387af0c512779d460e785b5db536c4"}
Nov 02 15:35:27.866325 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:27.866266    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg" podStartSLOduration=2.866250194 podStartE2EDuration="2.866250194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:35:27.86492736 +0000 UTC m=+2335.457544184" watchObservedRunningTime="2026-11-02 15:35:27.866250194 +0000 UTC m=+2335.458866992"
Nov 02 15:35:28.335242 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:28.335207    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:33.194782 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:33.194739    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"]
Nov 02 15:35:33.246225 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:33.246185    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-42555d-8pgmw"]
Nov 02 15:35:34.522209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:34.522177    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39a42c6-00da-437e-a9e2-b817223b1168" path="/var/lib/kubelet/pods/a39a42c6-00da-437e-a9e2-b817223b1168/volumes"
Nov 02 15:35:35.274828 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:35.274794    4891 scope.go:122] "RemoveContainer" containerID="443b33cdae15e6aaa0a8b1495ea37cbf456b95384cb4927f77423d7e1a9156c0"
Nov 02 15:35:35.285199 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:35.285176    4891 scope.go:122] "RemoveContainer" containerID="ebd08efbf85b008d70644dbdf1a38c0ee569a078f65c9ea5623cb9f6b6c128a7"
Nov 02 15:35:36.053027 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:36.052990    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"]
Nov 02 15:35:36.102423 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:36.102386    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2337f56a-d2nrj"]
Nov 02 15:35:36.105886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:36.105864    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"]
Nov 02 15:35:36.108739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:36.108720    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575229fb-pf5tt"]
Nov 02 15:35:36.335787 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:36.335702    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:36.348495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:36.348472    4891 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:36.524018 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:36.523985    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="749b1dac-26a8-41de-bfe9-89f861b7e10a" path="/var/lib/kubelet/pods/749b1dac-26a8-41de-bfe9-89f861b7e10a/volumes"
Nov 02 15:35:36.524496 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:36.524481    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f88f500a-1a7e-40fe-806d-b3b026ab2eb4" path="/var/lib/kubelet/pods/f88f500a-1a7e-40fe-806d-b3b026ab2eb4/volumes"
Nov 02 15:35:36.824890 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:36.824856    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:35:44.275066 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:35:44.275026    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:35:49.496489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:49.496452    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"]
Nov 02 15:35:49.545542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:49.545509    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-de35cd-lk5qw"]
Nov 02 15:35:50.522371 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:35:50.522335    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1734d55d-a7af-4691-a07f-c9c449e05dea" path="/var/lib/kubelet/pods/1734d55d-a7af-4691-a07f-c9c449e05dea/volumes"
Nov 02 15:36:12.048482 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.048442    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"]
Nov 02 15:36:12.052702 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.052680    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:12.113979 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.113944    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"]
Nov 02 15:36:12.158395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.158358    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:36:12.158395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.158387    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:36:12.191721 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.191685    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5\" (UID: \"6bdde05b-7bcc-48be-9a83-a47890f7567b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:12.191892 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.191754    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4pjt\" (UniqueName: \"kubernetes.io/projected/6bdde05b-7bcc-48be-9a83-a47890f7567b-kube-api-access-f4pjt\") pod \"maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5\" (UID: \"6bdde05b-7bcc-48be-9a83-a47890f7567b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:12.191892 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.191866    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5\" (UID: \"6bdde05b-7bcc-48be-9a83-a47890f7567b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:12.292778 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.292734    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5\" (UID: \"6bdde05b-7bcc-48be-9a83-a47890f7567b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:12.292967 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.292846    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5\" (UID: \"6bdde05b-7bcc-48be-9a83-a47890f7567b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:12.292967 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.292894    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4pjt\" (UniqueName: \"kubernetes.io/projected/6bdde05b-7bcc-48be-9a83-a47890f7567b-kube-api-access-f4pjt\") pod \"maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5\" (UID: \"6bdde05b-7bcc-48be-9a83-a47890f7567b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:12.293627 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.293601    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5\" (UID: \"6bdde05b-7bcc-48be-9a83-a47890f7567b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:12.295147 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.295128    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5\" (UID: \"6bdde05b-7bcc-48be-9a83-a47890f7567b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:12.354576 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.354500    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4pjt\" (UniqueName: \"kubernetes.io/projected/6bdde05b-7bcc-48be-9a83-a47890f7567b-kube-api-access-f4pjt\") pod \"maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5\" (UID: \"6bdde05b-7bcc-48be-9a83-a47890f7567b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:12.368396 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.368370    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:12.747938 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:12.747910    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"]
Nov 02 15:36:12.749218 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:36:12.749190    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bdde05b_7bcc_48be_9a83_a47890f7567b.slice/crio-1b2577d7ccc36627345a6eeff1baa08de3b8aab4e778f857daf053090068a6c4 WatchSource:0}: Error finding container 1b2577d7ccc36627345a6eeff1baa08de3b8aab4e778f857daf053090068a6c4: Status 404 returned error can't find the container with id 1b2577d7ccc36627345a6eeff1baa08de3b8aab4e778f857daf053090068a6c4
Nov 02 15:36:13.088213 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:13.088183    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5" event={"ID":"6bdde05b-7bcc-48be-9a83-a47890f7567b","Type":"ContainerStarted","Data":"e1f489f1413e998221c28aba3a92695cf36e8ddd57a430831e25b6a2d46027f5"}
Nov 02 15:36:13.088607 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:13.088237    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5" event={"ID":"6bdde05b-7bcc-48be-9a83-a47890f7567b","Type":"ContainerStarted","Data":"1b2577d7ccc36627345a6eeff1baa08de3b8aab4e778f857daf053090068a6c4"}
Nov 02 15:36:13.206258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:13.206198    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5" podStartSLOduration=1.206183653 podStartE2EDuration="1.206183653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:13.203052131 +0000 UTC m=+2380.795668932" watchObservedRunningTime="2026-11-02 15:36:13.206183653 +0000 UTC m=+2380.798800451"
Nov 02 15:36:14.097230 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:14.097196    4891 generic.go:356] "Generic (PLEG): container finished" podID="6bdde05b-7bcc-48be-9a83-a47890f7567b" containerID="e1f489f1413e998221c28aba3a92695cf36e8ddd57a430831e25b6a2d46027f5" exitCode=0
Nov 02 15:36:14.097659 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:14.097278    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5" event={"ID":"6bdde05b-7bcc-48be-9a83-a47890f7567b","Type":"ContainerDied","Data":"e1f489f1413e998221c28aba3a92695cf36e8ddd57a430831e25b6a2d46027f5"}
Nov 02 15:36:14.275290 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:36:14.275255    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:36:14.926435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:14.926398    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"]
Nov 02 15:36:14.926669 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:14.926621    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk" podUID="3f05246b-1509-4466-b06c-a3fb269b2ed6" containerName="istio-proxy" containerID="cri-o://f1504348560842303a24ea8874cfeb564c10515cd297dc7bfcbc4f8a1262e18f" gracePeriod=30
Nov 02 15:36:14.949633 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:14.949601    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"]
Nov 02 15:36:14.949812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:14.949793    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" podUID="02289c16-b904-4655-a895-125ee71ed905" containerName="istio-proxy" containerID="cri-o://bcc0fc03208a7b055961762ecfce8a7ac2f9492ee8792c3e0f801ffb9f14f328" gracePeriod=30
Nov 02 15:36:15.281284 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:15.281257    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:15.430216 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:15.430182    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-xks-ca\") pod \"6bdde05b-7bcc-48be-9a83-a47890f7567b\" (UID: \"6bdde05b-7bcc-48be-9a83-a47890f7567b\") "
Nov 02 15:36:15.430436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:15.430391    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-openshift-ca\") pod \"6bdde05b-7bcc-48be-9a83-a47890f7567b\" (UID: \"6bdde05b-7bcc-48be-9a83-a47890f7567b\") "
Nov 02 15:36:15.430511 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:15.430448    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bdde05b-7bcc-48be-9a83-a47890f7567b-kube-api-access-f4pjt\" (UniqueName: \"kubernetes.io/projected/6bdde05b-7bcc-48be-9a83-a47890f7567b-kube-api-access-f4pjt\") pod \"6bdde05b-7bcc-48be-9a83-a47890f7567b\" (UID: \"6bdde05b-7bcc-48be-9a83-a47890f7567b\") "
Nov 02 15:36:15.430886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:15.430854    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-openshift-ca" pod "6bdde05b-7bcc-48be-9a83-a47890f7567b" (UID: "6bdde05b-7bcc-48be-9a83-a47890f7567b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:15.431037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:15.430975    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:15.432410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:15.432381    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-xks-ca" pod "6bdde05b-7bcc-48be-9a83-a47890f7567b" (UID: "6bdde05b-7bcc-48be-9a83-a47890f7567b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:15.433057 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:15.433033    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bdde05b-7bcc-48be-9a83-a47890f7567b-kube-api-access-f4pjt" pod "6bdde05b-7bcc-48be-9a83-a47890f7567b" (UID: "6bdde05b-7bcc-48be-9a83-a47890f7567b"). InnerVolumeSpecName "kube-api-access-f4pjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:15.532365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:15.532336    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4pjt\" (UniqueName: \"kubernetes.io/projected/6bdde05b-7bcc-48be-9a83-a47890f7567b-kube-api-access-f4pjt\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:15.532365 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:15.532364    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bdde05b-7bcc-48be-9a83-a47890f7567b-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:16.114597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:16.114554    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5" event={"ID":"6bdde05b-7bcc-48be-9a83-a47890f7567b","Type":"ContainerDied","Data":"1b2577d7ccc36627345a6eeff1baa08de3b8aab4e778f857daf053090068a6c4"}
Nov 02 15:36:16.114755 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:16.114602    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"
Nov 02 15:36:16.114755 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:16.114651    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b2577d7ccc36627345a6eeff1baa08de3b8aab4e778f857daf053090068a6c4"
Nov 02 15:36:16.437263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:16.437183    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"]
Nov 02 15:36:16.444453 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:16.444424    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"]
Nov 02 15:36:16.458549 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:16.458522    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"]
Nov 02 15:36:17.051736 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:17.051702    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"]
Nov 02 15:36:17.056272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:17.056243    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1470ec-zdxvp"]
Nov 02 15:36:18.524873 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:18.524840    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b7dde3a-91d5-435a-a513-87ba52e835e5" path="/var/lib/kubelet/pods/7b7dde3a-91d5-435a-a513-87ba52e835e5/volumes"
Nov 02 15:36:20.154003 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.153973    4891 generic.go:356] "Generic (PLEG): container finished" podID="02289c16-b904-4655-a895-125ee71ed905" containerID="bcc0fc03208a7b055961762ecfce8a7ac2f9492ee8792c3e0f801ffb9f14f328" exitCode=0
Nov 02 15:36:20.154524 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.154130    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" event={"ID":"02289c16-b904-4655-a895-125ee71ed905","Type":"ContainerDied","Data":"bcc0fc03208a7b055961762ecfce8a7ac2f9492ee8792c3e0f801ffb9f14f328"}
Nov 02 15:36:20.156069 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.156034    4891 generic.go:356] "Generic (PLEG): container finished" podID="3f05246b-1509-4466-b06c-a3fb269b2ed6" containerID="f1504348560842303a24ea8874cfeb564c10515cd297dc7bfcbc4f8a1262e18f" exitCode=0
Nov 02 15:36:20.156235 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.156099    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk" event={"ID":"3f05246b-1509-4466-b06c-a3fb269b2ed6","Type":"ContainerDied","Data":"f1504348560842303a24ea8874cfeb564c10515cd297dc7bfcbc4f8a1262e18f"}
Nov 02 15:36:20.221697 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.221670    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:36:20.256853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.256828    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:36:20.287096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287062    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-istio-token\" (UniqueName: \"kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-istio-token\") pod \"02289c16-b904-4655-a895-125ee71ed905\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") "
Nov 02 15:36:20.287096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287103    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02289c16-b904-4655-a895-125ee71ed905-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02289c16-b904-4655-a895-125ee71ed905-istiod-ca-cert\") pod \"02289c16-b904-4655-a895-125ee71ed905\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") "
Nov 02 15:36:20.287334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287132    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-envoy\") pod \"02289c16-b904-4655-a895-125ee71ed905\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") "
Nov 02 15:36:20.287334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287242    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-data\") pod \"02289c16-b904-4655-a895-125ee71ed905\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") "
Nov 02 15:36:20.287334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287284    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-certs\") pod \"3f05246b-1509-4466-b06c-a3fb269b2ed6\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") "
Nov 02 15:36:20.287504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287369    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-certs\") pod \"02289c16-b904-4655-a895-125ee71ed905\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") "
Nov 02 15:36:20.287504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287420    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-socket\") pod \"3f05246b-1509-4466-b06c-a3fb269b2ed6\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") "
Nov 02 15:36:20.287504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287460    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-podinfo\") pod \"3f05246b-1509-4466-b06c-a3fb269b2ed6\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") "
Nov 02 15:36:20.287675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287540    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-token\" (UniqueName: \"kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-token\") pod \"3f05246b-1509-4466-b06c-a3fb269b2ed6\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") "
Nov 02 15:36:20.287675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287575    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-kube-api-access-b6kns\" (UniqueName: \"kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-kube-api-access-b6kns\") pod \"02289c16-b904-4655-a895-125ee71ed905\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") "
Nov 02 15:36:20.287675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287561    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-data" pod "02289c16-b904-4655-a895-125ee71ed905" (UID: "02289c16-b904-4655-a895-125ee71ed905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:20.287675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287648    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-certs" pod "3f05246b-1509-4466-b06c-a3fb269b2ed6" (UID: "3f05246b-1509-4466-b06c-a3fb269b2ed6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:20.287675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287658    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02289c16-b904-4655-a895-125ee71ed905-istiod-ca-cert" pod "02289c16-b904-4655-a895-125ee71ed905" (UID: "02289c16-b904-4655-a895-125ee71ed905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:20.287675 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287669    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-kube-api-access-d7kzr\" (UniqueName: \"kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-kube-api-access-d7kzr\") pod \"3f05246b-1509-4466-b06c-a3fb269b2ed6\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") "
Nov 02 15:36:20.288011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287711    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-data\") pod \"3f05246b-1509-4466-b06c-a3fb269b2ed6\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") "
Nov 02 15:36:20.288011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287750    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02289c16-b904-4655-a895-125ee71ed905-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02289c16-b904-4655-a895-125ee71ed905-istio-podinfo\") pod \"02289c16-b904-4655-a895-125ee71ed905\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") "
Nov 02 15:36:20.288011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287814    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-socket\") pod \"02289c16-b904-4655-a895-125ee71ed905\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") "
Nov 02 15:36:20.288011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287875    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-credential-socket\") pod \"02289c16-b904-4655-a895-125ee71ed905\" (UID: \"02289c16-b904-4655-a895-125ee71ed905\") "
Nov 02 15:36:20.288011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287908    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-envoy\") pod \"3f05246b-1509-4466-b06c-a3fb269b2ed6\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") "
Nov 02 15:36:20.288011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287943    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-socket" pod "3f05246b-1509-4466-b06c-a3fb269b2ed6" (UID: "3f05246b-1509-4466-b06c-a3fb269b2ed6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:20.288011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.287957    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-certs" pod "02289c16-b904-4655-a895-125ee71ed905" (UID: "02289c16-b904-4655-a895-125ee71ed905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:20.288447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.288179    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-socket" pod "02289c16-b904-4655-a895-125ee71ed905" (UID: "02289c16-b904-4655-a895-125ee71ed905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:20.288447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.288395    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-credential-socket" pod "02289c16-b904-4655-a895-125ee71ed905" (UID: "02289c16-b904-4655-a895-125ee71ed905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:20.290130 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290105    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-envoy" pod "02289c16-b904-4655-a895-125ee71ed905" (UID: "02289c16-b904-4655-a895-125ee71ed905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:20.290362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290290    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-kube-api-access-b6kns" pod "02289c16-b904-4655-a895-125ee71ed905" (UID: "02289c16-b904-4655-a895-125ee71ed905"). InnerVolumeSpecName "kube-api-access-b6kns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:20.290451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290361    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-data" pod "3f05246b-1509-4466-b06c-a3fb269b2ed6" (UID: "3f05246b-1509-4466-b06c-a3fb269b2ed6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:20.290451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290379    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f05246b-1509-4466-b06c-a3fb269b2ed6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f05246b-1509-4466-b06c-a3fb269b2ed6-istiod-ca-cert\") pod \"3f05246b-1509-4466-b06c-a3fb269b2ed6\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") "
Nov 02 15:36:20.290451 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290423    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-credential-socket\") pod \"3f05246b-1509-4466-b06c-a3fb269b2ed6\" (UID: \"3f05246b-1509-4466-b06c-a3fb269b2ed6\") "
Nov 02 15:36:20.290624 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290586    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-token" pod "3f05246b-1509-4466-b06c-a3fb269b2ed6" (UID: "3f05246b-1509-4466-b06c-a3fb269b2ed6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:20.290693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290634    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-istio-token" pod "02289c16-b904-4655-a895-125ee71ed905" (UID: "02289c16-b904-4655-a895-125ee71ed905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:20.290693 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290663    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-kube-api-access-d7kzr" pod "3f05246b-1509-4466-b06c-a3fb269b2ed6" (UID: "3f05246b-1509-4466-b06c-a3fb269b2ed6"). InnerVolumeSpecName "kube-api-access-d7kzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:20.290794 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290686    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-envoy" pod "3f05246b-1509-4466-b06c-a3fb269b2ed6" (UID: "3f05246b-1509-4466-b06c-a3fb269b2ed6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:20.290855 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290836    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-credential-socket" pod "3f05246b-1509-4466-b06c-a3fb269b2ed6" (UID: "3f05246b-1509-4466-b06c-a3fb269b2ed6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:20.290893 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290761    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f05246b-1509-4466-b06c-a3fb269b2ed6-istiod-ca-cert" pod "3f05246b-1509-4466-b06c-a3fb269b2ed6" (UID: "3f05246b-1509-4466-b06c-a3fb269b2ed6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:20.290992 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290976    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.290992    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291001    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291011    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291020    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6kns\" (UniqueName: \"kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-kube-api-access-b6kns\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291030    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7kzr\" (UniqueName: \"kubernetes.io/projected/3f05246b-1509-4466-b06c-a3fb269b2ed6-kube-api-access-d7kzr\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291038    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291047    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291055    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291062 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291065    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291073    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f05246b-1509-4466-b06c-a3fb269b2ed6-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291082    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f05246b-1509-4466-b06c-a3fb269b2ed6-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291090    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02289c16-b904-4655-a895-125ee71ed905-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291099    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02289c16-b904-4655-a895-125ee71ed905-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291109    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291038    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-podinfo" pod "3f05246b-1509-4466-b06c-a3fb269b2ed6" (UID: "3f05246b-1509-4466-b06c-a3fb269b2ed6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:36:20.291410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291122    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02289c16-b904-4655-a895-125ee71ed905-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.291410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.291159    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02289c16-b904-4655-a895-125ee71ed905-istio-podinfo" pod "02289c16-b904-4655-a895-125ee71ed905" (UID: "02289c16-b904-4655-a895-125ee71ed905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:36:20.391968 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.391935    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02289c16-b904-4655-a895-125ee71ed905-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:20.391968 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:20.391968    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f05246b-1509-4466-b06c-a3fb269b2ed6-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:21.165216 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:21.165184    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"
Nov 02 15:36:21.165732 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:21.165225    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf" event={"ID":"02289c16-b904-4655-a895-125ee71ed905","Type":"ContainerDied","Data":"033e68e3e525528147eea546c180416646f27dca4c756d5addaa5eaa083a8e56"}
Nov 02 15:36:21.165732 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:21.165284    4891 scope.go:122] "RemoveContainer" containerID="bcc0fc03208a7b055961762ecfce8a7ac2f9492ee8792c3e0f801ffb9f14f328"
Nov 02 15:36:21.166883 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:21.166864    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"
Nov 02 15:36:21.166979 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:21.166906    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk" event={"ID":"3f05246b-1509-4466-b06c-a3fb269b2ed6","Type":"ContainerDied","Data":"2bd291989977acc32a6ee38946cff6fbbf6300303d46d6891498fceb2f96eb4f"}
Nov 02 15:36:21.176200 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:21.176181    4891 scope.go:122] "RemoveContainer" containerID="f1504348560842303a24ea8874cfeb564c10515cd297dc7bfcbc4f8a1262e18f"
Nov 02 15:36:21.333006 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:21.332961    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"]
Nov 02 15:36:21.382222 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:21.382189    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-s87nf"]
Nov 02 15:36:21.446750 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:21.446717    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"]
Nov 02 15:36:21.452138 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:21.452107    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-de9c4ba4-openshift-default-854f5f846f-2sdrk"]
Nov 02 15:36:22.523134 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:22.523100    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02289c16-b904-4655-a895-125ee71ed905" path="/var/lib/kubelet/pods/02289c16-b904-4655-a895-125ee71ed905/volumes"
Nov 02 15:36:22.523871 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:22.523849    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f05246b-1509-4466-b06c-a3fb269b2ed6" path="/var/lib/kubelet/pods/3f05246b-1509-4466-b06c-a3fb269b2ed6/volumes"
Nov 02 15:36:31.460284 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.460238    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" containerID="cri-o://375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7" gracePeriod=45
Nov 02 15:36:31.464788 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.464755    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": EOF"
Nov 02 15:36:31.466715 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.466673    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main" containerID="cri-o://3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710" gracePeriod=45
Nov 02 15:36:31.480649 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.480612    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" containerID="cri-o://ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2" gracePeriod=45
Nov 02 15:36:31.524242 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.524201    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": dial tcp 10.132.0.54:8000: connect: connection refused"
Nov 02 15:36:31.775127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.775095    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:36:31.795692 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.795657    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:36:31.814740 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.814709    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:36:31.866435 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866402    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-dshm\") pod \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") "
Nov 02 15:36:31.866645 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866458    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-dshm\") pod \"f66fd185-3f21-4783-b080-72c2283e0152\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") "
Nov 02 15:36:31.866645 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866500    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f66fd185-3f21-4783-b080-72c2283e0152-tls-certs\" (UniqueName: \"kubernetes.io/secret/f66fd185-3f21-4783-b080-72c2283e0152-tls-certs\") pod \"f66fd185-3f21-4783-b080-72c2283e0152\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") "
Nov 02 15:36:31.866645 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866518    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-model-cache\") pod \"f66fd185-3f21-4783-b080-72c2283e0152\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") "
Nov 02 15:36:31.866645 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866545    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-home\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-home\") pod \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") "
Nov 02 15:36:31.866645 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866576    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-home\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-home\") pod \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") "
Nov 02 15:36:31.866953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866649    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-model-cache\") pod \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") "
Nov 02 15:36:31.866953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866691    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-dshm\") pod \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") "
Nov 02 15:36:31.866953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866749    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-home\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-home\") pod \"f66fd185-3f21-4783-b080-72c2283e0152\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") "
Nov 02 15:36:31.866953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866793    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tmp-dir\") pod \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") "
Nov 02 15:36:31.866953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866841    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tmp-dir\") pod \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") "
Nov 02 15:36:31.866953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866885    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66fd185-3f21-4783-b080-72c2283e0152-kube-api-access-qs6nl\" (UniqueName: \"kubernetes.io/projected/f66fd185-3f21-4783-b080-72c2283e0152-kube-api-access-qs6nl\") pod \"f66fd185-3f21-4783-b080-72c2283e0152\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") "
Nov 02 15:36:31.866953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866923    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tls-certs\") pod \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") "
Nov 02 15:36:31.866953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866954    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-tmp-dir\") pod \"f66fd185-3f21-4783-b080-72c2283e0152\" (UID: \"f66fd185-3f21-4783-b080-72c2283e0152\") "
Nov 02 15:36:31.867410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.866984    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-kube-api-access-pz9zj\" (UniqueName: \"kubernetes.io/projected/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-kube-api-access-pz9zj\") pod \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") "
Nov 02 15:36:31.867410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.867037    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tls-certs\") pod \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\" (UID: \"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4\") "
Nov 02 15:36:31.867410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.867088    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac326ed5-2b28-47d6-8ac2-216f41a29e45-kube-api-access-d9kwf\" (UniqueName: \"kubernetes.io/projected/ac326ed5-2b28-47d6-8ac2-216f41a29e45-kube-api-access-d9kwf\") pod \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") "
Nov 02 15:36:31.867410 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.867180    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-model-cache\") pod \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\" (UID: \"ac326ed5-2b28-47d6-8ac2-216f41a29e45\") "
Nov 02 15:36:31.867848 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.867820    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tmp-dir" pod "ac326ed5-2b28-47d6-8ac2-216f41a29e45" (UID: "ac326ed5-2b28-47d6-8ac2-216f41a29e45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:31.868079 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.867947    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-model-cache" pod "ac326ed5-2b28-47d6-8ac2-216f41a29e45" (UID: "ac326ed5-2b28-47d6-8ac2-216f41a29e45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:31.868219 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.868120    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tmp-dir" pod "9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" (UID: "9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:31.868348 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.868274    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-home" pod "ac326ed5-2b28-47d6-8ac2-216f41a29e45" (UID: "ac326ed5-2b28-47d6-8ac2-216f41a29e45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:31.868535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.868512    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-model-cache" pod "f66fd185-3f21-4783-b080-72c2283e0152" (UID: "f66fd185-3f21-4783-b080-72c2283e0152"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:31.868721 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.868701    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-model-cache" pod "9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" (UID: "9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:31.868827 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.868714    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-home" pod "9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" (UID: "9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:31.869619 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.869473    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66fd185-3f21-4783-b080-72c2283e0152-tls-certs" pod "f66fd185-3f21-4783-b080-72c2283e0152" (UID: "f66fd185-3f21-4783-b080-72c2283e0152"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:31.870183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.870126    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-home" pod "f66fd185-3f21-4783-b080-72c2283e0152" (UID: "f66fd185-3f21-4783-b080-72c2283e0152"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:31.870302 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.870175    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-dshm" pod "f66fd185-3f21-4783-b080-72c2283e0152" (UID: "f66fd185-3f21-4783-b080-72c2283e0152"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:31.870456 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.870396    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-tmp-dir" pod "f66fd185-3f21-4783-b080-72c2283e0152" (UID: "f66fd185-3f21-4783-b080-72c2283e0152"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:31.870586 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.870559    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66fd185-3f21-4783-b080-72c2283e0152-kube-api-access-qs6nl" pod "f66fd185-3f21-4783-b080-72c2283e0152" (UID: "f66fd185-3f21-4783-b080-72c2283e0152"). InnerVolumeSpecName "kube-api-access-qs6nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:31.871403 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.871376    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-dshm" pod "ac326ed5-2b28-47d6-8ac2-216f41a29e45" (UID: "ac326ed5-2b28-47d6-8ac2-216f41a29e45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:31.871525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.871399    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-dshm" pod "9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" (UID: "9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:31.871765 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.871749    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-kube-api-access-pz9zj" pod "9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" (UID: "9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4"). InnerVolumeSpecName "kube-api-access-pz9zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:31.871945 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.871916    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tls-certs" pod "ac326ed5-2b28-47d6-8ac2-216f41a29e45" (UID: "ac326ed5-2b28-47d6-8ac2-216f41a29e45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:31.872282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.872261    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tls-certs" pod "9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" (UID: "9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:31.873186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.873162    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac326ed5-2b28-47d6-8ac2-216f41a29e45-kube-api-access-d9kwf" pod "ac326ed5-2b28-47d6-8ac2-216f41a29e45" (UID: "ac326ed5-2b28-47d6-8ac2-216f41a29e45"). InnerVolumeSpecName "kube-api-access-d9kwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:31.968852 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968815    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.968852 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968845    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66fd185-3f21-4783-b080-72c2283e0152-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.968852 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968854    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968863    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968873    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968881    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968888    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968896    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968904    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968911    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968921    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs6nl\" (UniqueName: \"kubernetes.io/projected/f66fd185-3f21-4783-b080-72c2283e0152-kube-api-access-qs6nl\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968929    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac326ed5-2b28-47d6-8ac2-216f41a29e45-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968937    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66fd185-3f21-4783-b080-72c2283e0152-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968945    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz9zj\" (UniqueName: \"kubernetes.io/projected/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-kube-api-access-pz9zj\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968953    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968961    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9kwf\" (UniqueName: \"kubernetes.io/projected/ac326ed5-2b28-47d6-8ac2-216f41a29e45-kube-api-access-d9kwf\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968970    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac326ed5-2b28-47d6-8ac2-216f41a29e45-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:31.969093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:31.968978    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:32.315428 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.315386    4891 generic.go:356] "Generic (PLEG): container finished" podID="f66fd185-3f21-4783-b080-72c2283e0152" containerID="375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7" exitCode=0
Nov 02 15:36:32.315428 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.315424    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"
Nov 02 15:36:32.315647 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.315460    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" event={"ID":"f66fd185-3f21-4783-b080-72c2283e0152","Type":"ContainerDied","Data":"375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7"}
Nov 02 15:36:32.315647 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.315502    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq" event={"ID":"f66fd185-3f21-4783-b080-72c2283e0152","Type":"ContainerDied","Data":"a8ce802c37055524343225d57b2765ebde1b96459a6e0559475103603cba68ac"}
Nov 02 15:36:32.315647 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.315519    4891 scope.go:122] "RemoveContainer" containerID="375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7"
Nov 02 15:36:32.317254 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.317228    4891 generic.go:356] "Generic (PLEG): container finished" podID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerID="3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710" exitCode=0
Nov 02 15:36:32.317388 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.317281    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"
Nov 02 15:36:32.317388 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.317302    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" event={"ID":"ac326ed5-2b28-47d6-8ac2-216f41a29e45","Type":"ContainerDied","Data":"3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710"}
Nov 02 15:36:32.317388 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.317347    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm" event={"ID":"ac326ed5-2b28-47d6-8ac2-216f41a29e45","Type":"ContainerDied","Data":"5f55184bd4249941070b055e9d5abbc54ade3d323e640cea9bf3f58d6c807715"}
Nov 02 15:36:32.320272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.320247    4891 generic.go:356] "Generic (PLEG): container finished" podID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerID="ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2" exitCode=0
Nov 02 15:36:32.320436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.320293    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"
Nov 02 15:36:32.320436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.320365    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" event={"ID":"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4","Type":"ContainerDied","Data":"ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2"}
Nov 02 15:36:32.320436 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.320396    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b" event={"ID":"9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4","Type":"ContainerDied","Data":"4bd812c551323d47875a8f16c3a5f8f2b2b47871eb527631c607efac510a8b31"}
Nov 02 15:36:32.329939 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.329916    4891 scope.go:122] "RemoveContainer" containerID="375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7"
Nov 02 15:36:32.330264 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:36:32.330241    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7\": container with ID starting with 375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7 not found: ID does not exist" containerID="375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7"
Nov 02 15:36:32.330346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.330273    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7"} err="failed to get container status \"375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7\": rpc error: code = NotFound desc = could not find container \"375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7\": container with ID starting with 375cb0b6d7ce084b1a4b55b4f5d698e84b15f5fa2961c5c074611d988ec39dd7 not found: ID does not exist"
Nov 02 15:36:32.330346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.330291    4891 scope.go:122] "RemoveContainer" containerID="3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710"
Nov 02 15:36:32.340055 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.340038    4891 scope.go:122] "RemoveContainer" containerID="3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710"
Nov 02 15:36:32.340344 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:36:32.340321    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710\": container with ID starting with 3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710 not found: ID does not exist" containerID="3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710"
Nov 02 15:36:32.340443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.340349    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710"} err="failed to get container status \"3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710\": rpc error: code = NotFound desc = could not find container \"3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710\": container with ID starting with 3ceebe4a6f4eb22bdcbf800f2a948d8bbaaa95962cc07c348e2707f7576a1710 not found: ID does not exist"
Nov 02 15:36:32.340443 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.340366    4891 scope.go:122] "RemoveContainer" containerID="ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2"
Nov 02 15:36:32.352552 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.352531    4891 scope.go:122] "RemoveContainer" containerID="ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2"
Nov 02 15:36:32.352885 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:36:32.352864    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2\": container with ID starting with ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2 not found: ID does not exist" containerID="ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2"
Nov 02 15:36:32.352947 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.352894    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2"} err="failed to get container status \"ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2\": rpc error: code = NotFound desc = could not find container \"ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2\": container with ID starting with ba98e9a379d709842d77d93d8c1cc4bf621ccd24a7480540e0c20e30bfecf1e2 not found: ID does not exist"
Nov 02 15:36:32.486345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.486283    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"]
Nov 02 15:36:32.490956 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.490900    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/premium-w5-db3931-kserve-5fb7f878c5-dczxm"]
Nov 02 15:36:32.541360 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.540834    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" path="/var/lib/kubelet/pods/ac326ed5-2b28-47d6-8ac2-216f41a29e45/volumes"
Nov 02 15:36:32.607386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.606734    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"]
Nov 02 15:36:32.622334 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.619539    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/distinct-w5-db3931-kserve-7f95d5ddf8-7ghfq"]
Nov 02 15:36:32.742790 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.741583    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"]
Nov 02 15:36:32.768875 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:32.768827    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-db3931/simulator-w5-db3931-kserve-74c588ff66-n5h7b"]
Nov 02 15:36:33.230639 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:33.230610    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:36:33.232819 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:33.231978    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:36:33.234154 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:33.234133    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:36:33.235227 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:33.235207    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:36:33.253345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:33.253294    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:36:33.255630 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:33.255609    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:36:34.522392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:34.522356    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" path="/var/lib/kubelet/pods/9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4/volumes"
Nov 02 15:36:34.523002 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:34.522980    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66fd185-3f21-4783-b080-72c2283e0152" path="/var/lib/kubelet/pods/f66fd185-3f21-4783-b080-72c2283e0152/volumes"
Nov 02 15:36:35.337069 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:35.337030    4891 scope.go:122] "RemoveContainer" containerID="4b4493842e2c74ea36beefb3068ca2b3dd6a48191c0de8e86370e148c8c23bbc"
Nov 02 15:36:35.347192 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:35.347174    4891 scope.go:122] "RemoveContainer" containerID="21708c0a154988a955268b1985fcc337ccbc76ae3cc9645173f520d67986c04b"
Nov 02 15:36:35.356664 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:35.356642    4891 scope.go:122] "RemoveContainer" containerID="7deb9d8a695c7d22f70de72e7d230c8c477b66a66dcd2e42ec1d3645d5ad4368"
Nov 02 15:36:35.365688 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:35.365670    4891 scope.go:122] "RemoveContainer" containerID="6e9504e9d4dc1571bab3e9193dd667bbce5dc80c298d5ad30a9e25f5a17f748f"
Nov 02 15:36:35.374446 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:35.374428    4891 scope.go:122] "RemoveContainer" containerID="2d83c98643edb1a15525bdb6af61265d1c4832dad0a9780c7674d425f08e2b06"
Nov 02 15:36:35.383621 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:35.383601    4891 scope.go:122] "RemoveContainer" containerID="3131e484adf020d393e56e9c40de2ae2986bc798525e5409486dd5c9bc4f0a69"
Nov 02 15:36:35.392686 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:35.392664    4891 scope.go:122] "RemoveContainer" containerID="062f37198be3ab9ade81a94477bf720988c5460dff0d720f497b0e264d671fdd"
Nov 02 15:36:35.401480 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:35.401462    4891 scope.go:122] "RemoveContainer" containerID="720ddceda902a0f55880fe157176a73f6f1c29f7d89ee958b990fdba6f0be913"
Nov 02 15:36:39.057420 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.057383    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"]
Nov 02 15:36:39.057916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.057899    4891 cpu_manager.go:414] "containerMap: removing container" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main"
Nov 02 15:36:39.057916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.057918    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main"
Nov 02 15:36:39.058019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.057937    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main"
Nov 02 15:36:39.058019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.057943    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main"
Nov 02 15:36:39.058019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.057954    4891 cpu_manager.go:414] "containerMap: removing container" podUID="02289c16-b904-4655-a895-125ee71ed905" containerName="istio-proxy"
Nov 02 15:36:39.058019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.057959    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02289c16-b904-4655-a895-125ee71ed905" containerName="istio-proxy"
Nov 02 15:36:39.058019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.057974    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main"
Nov 02 15:36:39.058019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.057981    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main"
Nov 02 15:36:39.058019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.057988    4891 cpu_manager.go:414] "containerMap: removing container" podUID="3f05246b-1509-4466-b06c-a3fb269b2ed6" containerName="istio-proxy"
Nov 02 15:36:39.058019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.057994    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f05246b-1509-4466-b06c-a3fb269b2ed6" containerName="istio-proxy"
Nov 02 15:36:39.058019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.058001    4891 cpu_manager.go:414] "containerMap: removing container" podUID="6bdde05b-7bcc-48be-9a83-a47890f7567b" containerName="revoke-keys"
Nov 02 15:36:39.058019 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.058006    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bdde05b-7bcc-48be-9a83-a47890f7567b" containerName="revoke-keys"
Nov 02 15:36:39.058440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.058078    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac326ed5-2b28-47d6-8ac2-216f41a29e45" containerName="main"
Nov 02 15:36:39.058440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.058089    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66fd185-3f21-4783-b080-72c2283e0152" containerName="main"
Nov 02 15:36:39.058440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.058095    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="02289c16-b904-4655-a895-125ee71ed905" containerName="istio-proxy"
Nov 02 15:36:39.058440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.058102    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed60c0c-4ce2-42b3-b8a7-b2b0f2fa2cb4" containerName="main"
Nov 02 15:36:39.058440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.058108    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f05246b-1509-4466-b06c-a3fb269b2ed6" containerName="istio-proxy"
Nov 02 15:36:39.058440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.058114    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bdde05b-7bcc-48be-9a83-a47890f7567b" containerName="revoke-keys"
Nov 02 15:36:39.063291 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.063274    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:39.128231 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.128072    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:36:39.128484 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.128213    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:36:39.131042 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.131019    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"]
Nov 02 15:36:39.156007 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.155979    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1575e9c-44xkl\" (UID: \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:39.156161 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.156043    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlzbz\" (UniqueName: \"kubernetes.io/projected/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-kube-api-access-rlzbz\") pod \"maas-api-revoke-sub-f1575e9c-44xkl\" (UID: \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:39.156228 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.156180    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1575e9c-44xkl\" (UID: \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:39.257623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.257587    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlzbz\" (UniqueName: \"kubernetes.io/projected/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-kube-api-access-rlzbz\") pod \"maas-api-revoke-sub-f1575e9c-44xkl\" (UID: \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:39.257826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.257632    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1575e9c-44xkl\" (UID: \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:39.257826 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.257734    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1575e9c-44xkl\" (UID: \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:39.258535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.258508    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1575e9c-44xkl\" (UID: \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:39.260245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.260218    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1575e9c-44xkl\" (UID: \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:39.328245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.328169    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlzbz\" (UniqueName: \"kubernetes.io/projected/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-kube-api-access-rlzbz\") pod \"maas-api-revoke-sub-f1575e9c-44xkl\" (UID: \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:39.377171 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.377140    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:39.761734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:39.761703    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"]
Nov 02 15:36:39.763479 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:36:39.763448    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa7e1c5_5e70_4e7d_b8b7_6c81af30014e.slice/crio-12436418acbbf72e5b0e933a78de2bfe7982e32fd17185985e9f70101fc3c324 WatchSource:0}: Error finding container 12436418acbbf72e5b0e933a78de2bfe7982e32fd17185985e9f70101fc3c324: Status 404 returned error can't find the container with id 12436418acbbf72e5b0e933a78de2bfe7982e32fd17185985e9f70101fc3c324
Nov 02 15:36:40.385164 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:40.385071    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl" event={"ID":"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e","Type":"ContainerStarted","Data":"04b8a1578a5fa6ffea94a6b265964ba1b5108f13896fbc945fea3778db251119"}
Nov 02 15:36:40.385164 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:40.385122    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl" event={"ID":"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e","Type":"ContainerStarted","Data":"12436418acbbf72e5b0e933a78de2bfe7982e32fd17185985e9f70101fc3c324"}
Nov 02 15:36:40.503153 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:40.503098    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl" podStartSLOduration=2.503083667 podStartE2EDuration="2.503083667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:40.500558844 +0000 UTC m=+2408.093175640" watchObservedRunningTime="2026-11-02 15:36:40.503083667 +0000 UTC m=+2408.095700470"
Nov 02 15:36:41.393114 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:41.393082    4891 generic.go:356] "Generic (PLEG): container finished" podID="bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e" containerID="04b8a1578a5fa6ffea94a6b265964ba1b5108f13896fbc945fea3778db251119" exitCode=0
Nov 02 15:36:41.393538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:41.393164    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl" event={"ID":"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e","Type":"ContainerDied","Data":"04b8a1578a5fa6ffea94a6b265964ba1b5108f13896fbc945fea3778db251119"}
Nov 02 15:36:42.553599 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:42.553572    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:42.696830 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:42.696747    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-xks-ca\") pod \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\" (UID: \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\") "
Nov 02 15:36:42.696830 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:42.696817    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-kube-api-access-rlzbz\" (UniqueName: \"kubernetes.io/projected/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-kube-api-access-rlzbz\") pod \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\" (UID: \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\") "
Nov 02 15:36:42.697087 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:42.696896    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-openshift-ca\") pod \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\" (UID: \"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e\") "
Nov 02 15:36:42.697262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:42.697233    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-openshift-ca" pod "bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e" (UID: "bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:42.698886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:42.698862    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-xks-ca" pod "bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e" (UID: "bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:42.698974 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:42.698931    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-kube-api-access-rlzbz" pod "bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e" (UID: "bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e"). InnerVolumeSpecName "kube-api-access-rlzbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:42.798808 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:42.798769    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:42.798808 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:42.798800    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlzbz\" (UniqueName: \"kubernetes.io/projected/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-kube-api-access-rlzbz\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:42.798808 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:42.798812    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:36:43.409784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:43.409755    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"
Nov 02 15:36:43.409961 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:43.409791    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl" event={"ID":"bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e","Type":"ContainerDied","Data":"12436418acbbf72e5b0e933a78de2bfe7982e32fd17185985e9f70101fc3c324"}
Nov 02 15:36:43.409961 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:43.409839    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12436418acbbf72e5b0e933a78de2bfe7982e32fd17185985e9f70101fc3c324"
Nov 02 15:36:44.273396 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:36:44.273358    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:36:52.073006 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:52.072971    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"]
Nov 02 15:36:52.122629 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:52.122589    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"]
Nov 02 15:36:52.125906 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:52.125882    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c0fa4d-q7xn6"]
Nov 02 15:36:52.128743 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:52.128719    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a086b6ea-54nqx"]
Nov 02 15:36:52.522798 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:52.522769    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b1efe6-5e7d-4324-8bcb-b80686cdfbb3" path="/var/lib/kubelet/pods/31b1efe6-5e7d-4324-8bcb-b80686cdfbb3/volumes"
Nov 02 15:36:52.523220 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:36:52.523207    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1ae3e8-21e1-4283-8917-0600eafdddf5" path="/var/lib/kubelet/pods/db1ae3e8-21e1-4283-8917-0600eafdddf5/volumes"
Nov 02 15:37:13.468990 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.468953    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"]
Nov 02 15:37:13.469490 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.469475    4891 cpu_manager.go:414] "containerMap: removing container" podUID="bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e" containerName="revoke-keys"
Nov 02 15:37:13.469548 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.469491    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e" containerName="revoke-keys"
Nov 02 15:37:13.469589 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.469580    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e" containerName="revoke-keys"
Nov 02 15:37:13.472891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.472876    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:13.531564 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.531533    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"]
Nov 02 15:37:13.543049 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.543015    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c3bb5b8-lv249\" (UID: \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:13.543230 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.543203    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6prlh\" (UniqueName: \"kubernetes.io/projected/9d75071d-68cc-47e2-ae69-2adfb61f61c6-kube-api-access-6prlh\") pod \"maas-api-revoke-sub-1c3bb5b8-lv249\" (UID: \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:13.543383 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.543360    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c3bb5b8-lv249\" (UID: \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:13.576429 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.576402    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:37:13.576671 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.576650    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:37:13.645022 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.644972    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6prlh\" (UniqueName: \"kubernetes.io/projected/9d75071d-68cc-47e2-ae69-2adfb61f61c6-kube-api-access-6prlh\") pod \"maas-api-revoke-sub-1c3bb5b8-lv249\" (UID: \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:13.645222 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.645083    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c3bb5b8-lv249\" (UID: \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:13.645303 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.645280    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c3bb5b8-lv249\" (UID: \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:13.645863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.645835    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c3bb5b8-lv249\" (UID: \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:13.647746 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.647724    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c3bb5b8-lv249\" (UID: \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:13.706448 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.706423    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6prlh\" (UniqueName: \"kubernetes.io/projected/9d75071d-68cc-47e2-ae69-2adfb61f61c6-kube-api-access-6prlh\") pod \"maas-api-revoke-sub-1c3bb5b8-lv249\" (UID: \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:13.785919 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:13.785885    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:14.169606 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:14.169573    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"]
Nov 02 15:37:14.170632 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:37:14.170610    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d75071d_68cc_47e2_ae69_2adfb61f61c6.slice/crio-71fa3985e50ae893531f0df0645d2cca6ba32aea551f6c650e20a5af9a391f34 WatchSource:0}: Error finding container 71fa3985e50ae893531f0df0645d2cca6ba32aea551f6c650e20a5af9a391f34: Status 404 returned error can't find the container with id 71fa3985e50ae893531f0df0645d2cca6ba32aea551f6c650e20a5af9a391f34
Nov 02 15:37:14.370272 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:37:14.370233    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:37:14.660790 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:14.660758    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249" event={"ID":"9d75071d-68cc-47e2-ae69-2adfb61f61c6","Type":"ContainerStarted","Data":"9173ce2d0e34a28a84ea0ad6da67fabf47832cc5db343133481c6556dfdc271c"}
Nov 02 15:37:14.661179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:14.660807    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249" event={"ID":"9d75071d-68cc-47e2-ae69-2adfb61f61c6","Type":"ContainerStarted","Data":"71fa3985e50ae893531f0df0645d2cca6ba32aea551f6c650e20a5af9a391f34"}
Nov 02 15:37:14.777307 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:14.777255    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249" podStartSLOduration=1.777240558 podStartE2EDuration="1.777240558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:14.776265876 +0000 UTC m=+2442.368882675" watchObservedRunningTime="2026-11-02 15:37:14.777240558 +0000 UTC m=+2442.369857356"
Nov 02 15:37:15.668832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:15.668797    4891 generic.go:356] "Generic (PLEG): container finished" podID="9d75071d-68cc-47e2-ae69-2adfb61f61c6" containerID="9173ce2d0e34a28a84ea0ad6da67fabf47832cc5db343133481c6556dfdc271c" exitCode=0
Nov 02 15:37:15.669234 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:15.668875    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249" event={"ID":"9d75071d-68cc-47e2-ae69-2adfb61f61c6","Type":"ContainerDied","Data":"9173ce2d0e34a28a84ea0ad6da67fabf47832cc5db343133481c6556dfdc271c"}
Nov 02 15:37:16.834553 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:16.834528    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:16.980011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:16.979930    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-xks-ca\") pod \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\" (UID: \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\") "
Nov 02 15:37:16.980011 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:16.979969    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-openshift-ca\") pod \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\" (UID: \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\") "
Nov 02 15:37:16.980253 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:16.980199    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d75071d-68cc-47e2-ae69-2adfb61f61c6-kube-api-access-6prlh\" (UniqueName: \"kubernetes.io/projected/9d75071d-68cc-47e2-ae69-2adfb61f61c6-kube-api-access-6prlh\") pod \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\" (UID: \"9d75071d-68cc-47e2-ae69-2adfb61f61c6\") "
Nov 02 15:37:16.980521 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:16.980487    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-openshift-ca" pod "9d75071d-68cc-47e2-ae69-2adfb61f61c6" (UID: "9d75071d-68cc-47e2-ae69-2adfb61f61c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:16.980668 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:16.980647    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:16.982160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:16.982134    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-xks-ca" pod "9d75071d-68cc-47e2-ae69-2adfb61f61c6" (UID: "9d75071d-68cc-47e2-ae69-2adfb61f61c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:16.982408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:16.982381    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d75071d-68cc-47e2-ae69-2adfb61f61c6-kube-api-access-6prlh" pod "9d75071d-68cc-47e2-ae69-2adfb61f61c6" (UID: "9d75071d-68cc-47e2-ae69-2adfb61f61c6"). InnerVolumeSpecName "kube-api-access-6prlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:17.082026 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:17.081990    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6prlh\" (UniqueName: \"kubernetes.io/projected/9d75071d-68cc-47e2-ae69-2adfb61f61c6-kube-api-access-6prlh\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:17.082026 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:17.082022    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d75071d-68cc-47e2-ae69-2adfb61f61c6-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:17.690801 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:17.690771    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"
Nov 02 15:37:17.691001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:17.690819    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249" event={"ID":"9d75071d-68cc-47e2-ae69-2adfb61f61c6","Type":"ContainerDied","Data":"71fa3985e50ae893531f0df0645d2cca6ba32aea551f6c650e20a5af9a391f34"}
Nov 02 15:37:17.691001 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:17.690878    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71fa3985e50ae893531f0df0645d2cca6ba32aea551f6c650e20a5af9a391f34"
Nov 02 15:37:20.734852 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:20.734815    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"]
Nov 02 15:37:20.737822 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:20.737799    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-db3931-p75n5"]
Nov 02 15:37:22.533816 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:22.533785    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bdde05b-7bcc-48be-9a83-a47890f7567b" path="/var/lib/kubelet/pods/6bdde05b-7bcc-48be-9a83-a47890f7567b/volumes"
Nov 02 15:37:24.719270 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.719189    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"]
Nov 02 15:37:24.719716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.719691    4891 cpu_manager.go:414] "containerMap: removing container" podUID="9d75071d-68cc-47e2-ae69-2adfb61f61c6" containerName="revoke-keys"
Nov 02 15:37:24.719716 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.719704    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d75071d-68cc-47e2-ae69-2adfb61f61c6" containerName="revoke-keys"
Nov 02 15:37:24.719837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.719823    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d75071d-68cc-47e2-ae69-2adfb61f61c6" containerName="revoke-keys"
Nov 02 15:37:24.724187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.724169    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:24.777545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.777517    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:37:24.826597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.826568    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:37:24.830184 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.830151    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"]
Nov 02 15:37:24.874902 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.874873    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g\" (UID: \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:24.875060 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.875047    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g\" (UID: \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:24.875117 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.875086    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6z7q\" (UniqueName: \"kubernetes.io/projected/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-kube-api-access-w6z7q\") pod \"maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g\" (UID: \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:24.976708 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.976615    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g\" (UID: \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:24.976891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.976745    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g\" (UID: \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:24.976891 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.976772    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6z7q\" (UniqueName: \"kubernetes.io/projected/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-kube-api-access-w6z7q\") pod \"maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g\" (UID: \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:24.977390 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.977365    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g\" (UID: \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:24.979099 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:24.979077    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g\" (UID: \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:25.038103 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:25.038069    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6z7q\" (UniqueName: \"kubernetes.io/projected/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-kube-api-access-w6z7q\") pod \"maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g\" (UID: \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:25.337994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:25.337962    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:25.729474 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:25.729255    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"]
Nov 02 15:37:25.732065 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:37:25.732031    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c2c23b_a1d8_4ce0_b962_92d1aaa43c44.slice/crio-78215cdc40cacc355242e73639d7e967a800b5c1faf0e7a9ebd7f89487254f9b WatchSource:0}: Error finding container 78215cdc40cacc355242e73639d7e967a800b5c1faf0e7a9ebd7f89487254f9b: Status 404 returned error can't find the container with id 78215cdc40cacc355242e73639d7e967a800b5c1faf0e7a9ebd7f89487254f9b
Nov 02 15:37:25.752392 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:25.752364    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g" event={"ID":"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44","Type":"ContainerStarted","Data":"78215cdc40cacc355242e73639d7e967a800b5c1faf0e7a9ebd7f89487254f9b"}
Nov 02 15:37:26.761242 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:26.761211    4891 generic.go:356] "Generic (PLEG): container finished" podID="f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44" containerID="98974306c7bb4cfa666d5449a30a52f332318f373cac3b10b1882a75b679f120" exitCode=0
Nov 02 15:37:26.761677 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:26.761294    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g" event={"ID":"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44","Type":"ContainerDied","Data":"98974306c7bb4cfa666d5449a30a52f332318f373cac3b10b1882a75b679f120"}
Nov 02 15:37:27.920136 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:27.920101    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:28.011211 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.011178    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-openshift-ca\") pod \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\" (UID: \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\") "
Nov 02 15:37:28.011424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.011375    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-kube-api-access-w6z7q\" (UniqueName: \"kubernetes.io/projected/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-kube-api-access-w6z7q\") pod \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\" (UID: \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\") "
Nov 02 15:37:28.011513 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.011496    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-xks-ca\") pod \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\" (UID: \"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44\") "
Nov 02 15:37:28.011626 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.011605    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-openshift-ca" pod "f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44" (UID: "f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:28.011842 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.011819    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:28.013488 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.013459    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-kube-api-access-w6z7q" pod "f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44" (UID: "f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44"). InnerVolumeSpecName "kube-api-access-w6z7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:28.014106 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.014076    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-xks-ca" pod "f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44" (UID: "f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:28.113276 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.113239    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6z7q\" (UniqueName: \"kubernetes.io/projected/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-kube-api-access-w6z7q\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:28.113276 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.113270    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:28.150385 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.150301    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"]
Nov 02 15:37:28.152152 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.152115    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" podUID="7304fe30-39ac-4e9a-b880-20cc50287018" containerName="istio-proxy" containerID="cri-o://72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5" gracePeriod=30
Nov 02 15:37:28.776101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.776073    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"
Nov 02 15:37:28.776274 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.776109    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g" event={"ID":"f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44","Type":"ContainerDied","Data":"78215cdc40cacc355242e73639d7e967a800b5c1faf0e7a9ebd7f89487254f9b"}
Nov 02 15:37:28.776274 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:28.776155    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78215cdc40cacc355242e73639d7e967a800b5c1faf0e7a9ebd7f89487254f9b"
Nov 02 15:37:29.186119 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:29.186086    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"]
Nov 02 15:37:29.194593 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:29.194543    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"]
Nov 02 15:37:29.208481 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:29.208435    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"]
Nov 02 15:37:33.416690 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.416625    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:37:33.469093 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469064    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-certs\") pod \"7304fe30-39ac-4e9a-b880-20cc50287018\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") "
Nov 02 15:37:33.469273 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469122    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-socket\") pod \"7304fe30-39ac-4e9a-b880-20cc50287018\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") "
Nov 02 15:37:33.469273 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469176    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-credential-socket\") pod \"7304fe30-39ac-4e9a-b880-20cc50287018\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") "
Nov 02 15:37:33.469273 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469206    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-istio-token\" (UniqueName: \"kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-istio-token\") pod \"7304fe30-39ac-4e9a-b880-20cc50287018\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") "
Nov 02 15:37:33.469273 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469232    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7304fe30-39ac-4e9a-b880-20cc50287018-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7304fe30-39ac-4e9a-b880-20cc50287018-istiod-ca-cert\") pod \"7304fe30-39ac-4e9a-b880-20cc50287018\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") "
Nov 02 15:37:33.469532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469305    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-kube-api-access-55f4s\" (UniqueName: \"kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-kube-api-access-55f4s\") pod \"7304fe30-39ac-4e9a-b880-20cc50287018\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") "
Nov 02 15:37:33.469532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469368    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-certs" pod "7304fe30-39ac-4e9a-b880-20cc50287018" (UID: "7304fe30-39ac-4e9a-b880-20cc50287018"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:33.469532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469385    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-data\") pod \"7304fe30-39ac-4e9a-b880-20cc50287018\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") "
Nov 02 15:37:33.469532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469451    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7304fe30-39ac-4e9a-b880-20cc50287018-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7304fe30-39ac-4e9a-b880-20cc50287018-istio-podinfo\") pod \"7304fe30-39ac-4e9a-b880-20cc50287018\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") "
Nov 02 15:37:33.469532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469481    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-socket" pod "7304fe30-39ac-4e9a-b880-20cc50287018" (UID: "7304fe30-39ac-4e9a-b880-20cc50287018"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:33.469532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469523    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-credential-socket" pod "7304fe30-39ac-4e9a-b880-20cc50287018" (UID: "7304fe30-39ac-4e9a-b880-20cc50287018"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:33.469532 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469516    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-envoy\") pod \"7304fe30-39ac-4e9a-b880-20cc50287018\" (UID: \"7304fe30-39ac-4e9a-b880-20cc50287018\") "
Nov 02 15:37:33.469873 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469636    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-data" pod "7304fe30-39ac-4e9a-b880-20cc50287018" (UID: "7304fe30-39ac-4e9a-b880-20cc50287018"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:33.469873 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.469737    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7304fe30-39ac-4e9a-b880-20cc50287018-istiod-ca-cert" pod "7304fe30-39ac-4e9a-b880-20cc50287018" (UID: "7304fe30-39ac-4e9a-b880-20cc50287018"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:33.470342 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.470307    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:33.470433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.470348    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:33.470433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.470363    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:33.470433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.470377    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7304fe30-39ac-4e9a-b880-20cc50287018-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:33.470433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.470392    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:33.471623 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.471591    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7304fe30-39ac-4e9a-b880-20cc50287018-istio-podinfo" pod "7304fe30-39ac-4e9a-b880-20cc50287018" (UID: "7304fe30-39ac-4e9a-b880-20cc50287018"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:37:33.471768 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.471667    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-envoy" pod "7304fe30-39ac-4e9a-b880-20cc50287018" (UID: "7304fe30-39ac-4e9a-b880-20cc50287018"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:33.471966 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.471947    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-kube-api-access-55f4s" pod "7304fe30-39ac-4e9a-b880-20cc50287018" (UID: "7304fe30-39ac-4e9a-b880-20cc50287018"). InnerVolumeSpecName "kube-api-access-55f4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:33.472032 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.471992    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-istio-token" pod "7304fe30-39ac-4e9a-b880-20cc50287018" (UID: "7304fe30-39ac-4e9a-b880-20cc50287018"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:33.571810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.571772    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55f4s\" (UniqueName: \"kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-kube-api-access-55f4s\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:33.571810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.571806    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7304fe30-39ac-4e9a-b880-20cc50287018-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:33.572035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.571820    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7304fe30-39ac-4e9a-b880-20cc50287018-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:33.572035 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.571833    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7304fe30-39ac-4e9a-b880-20cc50287018-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:33.812837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.812803    4891 generic.go:356] "Generic (PLEG): container finished" podID="7304fe30-39ac-4e9a-b880-20cc50287018" containerID="72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5" exitCode=0
Nov 02 15:37:33.813052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.812842    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" event={"ID":"7304fe30-39ac-4e9a-b880-20cc50287018","Type":"ContainerDied","Data":"72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5"}
Nov 02 15:37:33.813052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.812882    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"
Nov 02 15:37:33.813052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.812907    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr" event={"ID":"7304fe30-39ac-4e9a-b880-20cc50287018","Type":"ContainerDied","Data":"4b40132c748387d5d62b67ea89e837dab49514d17eb0e86054aa8e5ddfb0fe82"}
Nov 02 15:37:33.813052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.812947    4891 scope.go:122] "RemoveContainer" containerID="72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5"
Nov 02 15:37:33.828491 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.828471    4891 scope.go:122] "RemoveContainer" containerID="72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5"
Nov 02 15:37:33.828984 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:37:33.828962    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5\": container with ID starting with 72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5 not found: ID does not exist" containerID="72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5"
Nov 02 15:37:33.829089 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.828990    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5"} err="failed to get container status \"72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5\": rpc error: code = NotFound desc = could not find container \"72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5\": container with ID starting with 72447c91f690281a66069cf13a4b9ee29e75f1369ba6b0cfcf1d9d1456b702c5 not found: ID does not exist"
Nov 02 15:37:33.979267 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.979235    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"]
Nov 02 15:37:33.982285 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:33.982254    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db3931-openshift-default-5f47d69f9f-x5xvr"]
Nov 02 15:37:34.522588 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:34.522549    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7304fe30-39ac-4e9a-b880-20cc50287018" path="/var/lib/kubelet/pods/7304fe30-39ac-4e9a-b880-20cc50287018/volumes"
Nov 02 15:37:35.487739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:35.487702    4891 scope.go:122] "RemoveContainer" containerID="4df5f2ae70ac78c89a1944c3e685efe7c3e6ee403a2899ebb821416853c3ce6c"
Nov 02 15:37:35.497417 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:35.497393    4891 scope.go:122] "RemoveContainer" containerID="b9fcdd15c5ac21ab65f8b0b58b7554b57683d66f92c649ced1e087f755e9f185"
Nov 02 15:37:35.506930 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:35.506900    4891 scope.go:122] "RemoveContainer" containerID="21f696804bc9820140eb34cfe3cc64a353ab5d3e987a4e43df3a321b7f3aa695"
Nov 02 15:37:35.516729 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:35.516706    4891 scope.go:122] "RemoveContainer" containerID="761ab4d28830a6d9900aa83819268f8182cd3d64ce028a79db7503ca6b7f873d"
Nov 02 15:37:36.330718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:36.330684    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"]
Nov 02 15:37:41.331142 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:41.331098    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26" podUID="08e8a676-324b-450f-85b2-8ec2df940dae" containerName="payload-pre-processing" containerID="cri-o://158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88" gracePeriod=25
Nov 02 15:37:44.206746 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.206659    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" containerID="cri-o://8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968" gracePeriod=45
Nov 02 15:37:44.213935 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.213887    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" containerID="cri-o://5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7" gracePeriod=45
Nov 02 15:37:44.234221 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.234188    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main" containerID="cri-o://c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8" gracePeriod=45
Nov 02 15:37:44.274369 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:37:44.274284    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:37:44.520187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.520157    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:37:44.522690 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.522662    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:37:44.544075 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.544044    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:37:44.623163 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623124    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-home\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-home\") pod \"2dbba973-9a21-4bed-9112-e9d0f9282d68\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") "
Nov 02 15:37:44.623353 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623190    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dbba973-9a21-4bed-9112-e9d0f9282d68-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbba973-9a21-4bed-9112-e9d0f9282d68-tls-certs\") pod \"2dbba973-9a21-4bed-9112-e9d0f9282d68\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") "
Nov 02 15:37:44.623353 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623226    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d2092c-52f2-499d-85f7-8681595ea6b3-kube-api-access-x9zf2\" (UniqueName: \"kubernetes.io/projected/73d2092c-52f2-499d-85f7-8681595ea6b3-kube-api-access-x9zf2\") pod \"73d2092c-52f2-499d-85f7-8681595ea6b3\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") "
Nov 02 15:37:44.623353 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623263    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dbba973-9a21-4bed-9112-e9d0f9282d68-kube-api-access-xbhxl\" (UniqueName: \"kubernetes.io/projected/2dbba973-9a21-4bed-9112-e9d0f9282d68-kube-api-access-xbhxl\") pod \"2dbba973-9a21-4bed-9112-e9d0f9282d68\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") "
Nov 02 15:37:44.623353 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623291    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-model-cache\") pod \"2dbba973-9a21-4bed-9112-e9d0f9282d68\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") "
Nov 02 15:37:44.623353 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623345    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75845278-3169-49ce-ace9-40a8a6ee001a-kube-api-access-bc69k\" (UniqueName: \"kubernetes.io/projected/75845278-3169-49ce-ace9-40a8a6ee001a-kube-api-access-bc69k\") pod \"75845278-3169-49ce-ace9-40a8a6ee001a\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") "
Nov 02 15:37:44.623666 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623413    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-home" pod "2dbba973-9a21-4bed-9112-e9d0f9282d68" (UID: "2dbba973-9a21-4bed-9112-e9d0f9282d68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:44.623666 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623621    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-model-cache" pod "2dbba973-9a21-4bed-9112-e9d0f9282d68" (UID: "2dbba973-9a21-4bed-9112-e9d0f9282d68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:44.623789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623695    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-dshm\") pod \"73d2092c-52f2-499d-85f7-8681595ea6b3\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") "
Nov 02 15:37:44.623789 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623764    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-dshm\") pod \"75845278-3169-49ce-ace9-40a8a6ee001a\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") "
Nov 02 15:37:44.623910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623799    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75845278-3169-49ce-ace9-40a8a6ee001a-tls-certs\" (UniqueName: \"kubernetes.io/secret/75845278-3169-49ce-ace9-40a8a6ee001a-tls-certs\") pod \"75845278-3169-49ce-ace9-40a8a6ee001a\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") "
Nov 02 15:37:44.623910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623841    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-dshm\") pod \"2dbba973-9a21-4bed-9112-e9d0f9282d68\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") "
Nov 02 15:37:44.623910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623875    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-model-cache\") pod \"75845278-3169-49ce-ace9-40a8a6ee001a\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") "
Nov 02 15:37:44.624077 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.623966    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73d2092c-52f2-499d-85f7-8681595ea6b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/73d2092c-52f2-499d-85f7-8681595ea6b3-tls-certs\") pod \"73d2092c-52f2-499d-85f7-8681595ea6b3\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") "
Nov 02 15:37:44.624077 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.624025    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-model-cache\") pod \"73d2092c-52f2-499d-85f7-8681595ea6b3\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") "
Nov 02 15:37:44.624198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.624078    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-home\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-home\") pod \"73d2092c-52f2-499d-85f7-8681595ea6b3\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") "
Nov 02 15:37:44.624198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.624116    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-home\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-home\") pod \"75845278-3169-49ce-ace9-40a8a6ee001a\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") "
Nov 02 15:37:44.624198 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.624160    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-tmp-dir\") pod \"73d2092c-52f2-499d-85f7-8681595ea6b3\" (UID: \"73d2092c-52f2-499d-85f7-8681595ea6b3\") "
Nov 02 15:37:44.624401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.624214    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-tmp-dir\") pod \"2dbba973-9a21-4bed-9112-e9d0f9282d68\" (UID: \"2dbba973-9a21-4bed-9112-e9d0f9282d68\") "
Nov 02 15:37:44.624401 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.624282    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-tmp-dir\") pod \"75845278-3169-49ce-ace9-40a8a6ee001a\" (UID: \"75845278-3169-49ce-ace9-40a8a6ee001a\") "
Nov 02 15:37:44.625878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.624784    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.625878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.624809    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.625878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.625046    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-tmp-dir" pod "75845278-3169-49ce-ace9-40a8a6ee001a" (UID: "75845278-3169-49ce-ace9-40a8a6ee001a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:44.625878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.625112    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-home" pod "75845278-3169-49ce-ace9-40a8a6ee001a" (UID: "75845278-3169-49ce-ace9-40a8a6ee001a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:44.625878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.625267    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-tmp-dir" pod "73d2092c-52f2-499d-85f7-8681595ea6b3" (UID: "73d2092c-52f2-499d-85f7-8681595ea6b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:44.625878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.625366    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-model-cache" pod "73d2092c-52f2-499d-85f7-8681595ea6b3" (UID: "73d2092c-52f2-499d-85f7-8681595ea6b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:44.625878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.625594    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-tmp-dir" pod "2dbba973-9a21-4bed-9112-e9d0f9282d68" (UID: "2dbba973-9a21-4bed-9112-e9d0f9282d68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:44.625878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.625631    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-home" pod "73d2092c-52f2-499d-85f7-8681595ea6b3" (UID: "73d2092c-52f2-499d-85f7-8681595ea6b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:44.625878 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.625800    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dbba973-9a21-4bed-9112-e9d0f9282d68-tls-certs" pod "2dbba973-9a21-4bed-9112-e9d0f9282d68" (UID: "2dbba973-9a21-4bed-9112-e9d0f9282d68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:44.626953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.625897    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d2092c-52f2-499d-85f7-8681595ea6b3-kube-api-access-x9zf2" pod "73d2092c-52f2-499d-85f7-8681595ea6b3" (UID: "73d2092c-52f2-499d-85f7-8681595ea6b3"). InnerVolumeSpecName "kube-api-access-x9zf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:44.626953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.625976    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbba973-9a21-4bed-9112-e9d0f9282d68-kube-api-access-xbhxl" pod "2dbba973-9a21-4bed-9112-e9d0f9282d68" (UID: "2dbba973-9a21-4bed-9112-e9d0f9282d68"). InnerVolumeSpecName "kube-api-access-xbhxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:44.626953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.626075    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-model-cache" pod "75845278-3169-49ce-ace9-40a8a6ee001a" (UID: "75845278-3169-49ce-ace9-40a8a6ee001a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:44.626953 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.626444    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75845278-3169-49ce-ace9-40a8a6ee001a-kube-api-access-bc69k" pod "75845278-3169-49ce-ace9-40a8a6ee001a" (UID: "75845278-3169-49ce-ace9-40a8a6ee001a"). InnerVolumeSpecName "kube-api-access-bc69k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:44.627750 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.627727    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-dshm" pod "73d2092c-52f2-499d-85f7-8681595ea6b3" (UID: "73d2092c-52f2-499d-85f7-8681595ea6b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:44.627833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.627810    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75845278-3169-49ce-ace9-40a8a6ee001a-tls-certs" pod "75845278-3169-49ce-ace9-40a8a6ee001a" (UID: "75845278-3169-49ce-ace9-40a8a6ee001a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:44.627973 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.627948    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-dshm" pod "2dbba973-9a21-4bed-9112-e9d0f9282d68" (UID: "2dbba973-9a21-4bed-9112-e9d0f9282d68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:44.628042 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.628022    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d2092c-52f2-499d-85f7-8681595ea6b3-tls-certs" pod "73d2092c-52f2-499d-85f7-8681595ea6b3" (UID: "73d2092c-52f2-499d-85f7-8681595ea6b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:44.628170 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.628155    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-dshm" pod "75845278-3169-49ce-ace9-40a8a6ee001a" (UID: "75845278-3169-49ce-ace9-40a8a6ee001a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:44.725750 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725718    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725750 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725747    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725750 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725759    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73d2092c-52f2-499d-85f7-8681595ea6b3-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725769    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725777    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725785    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725792    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725801    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbba973-9a21-4bed-9112-e9d0f9282d68-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725808    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725815    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbba973-9a21-4bed-9112-e9d0f9282d68-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725824    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9zf2\" (UniqueName: \"kubernetes.io/projected/73d2092c-52f2-499d-85f7-8681595ea6b3-kube-api-access-x9zf2\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725834    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbhxl\" (UniqueName: \"kubernetes.io/projected/2dbba973-9a21-4bed-9112-e9d0f9282d68-kube-api-access-xbhxl\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725842    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc69k\" (UniqueName: \"kubernetes.io/projected/75845278-3169-49ce-ace9-40a8a6ee001a-kube-api-access-bc69k\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725850    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73d2092c-52f2-499d-85f7-8681595ea6b3-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725857    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75845278-3169-49ce-ace9-40a8a6ee001a-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.725991 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.725864    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75845278-3169-49ce-ace9-40a8a6ee001a-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:44.946283 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.946247    4891 generic.go:356] "Generic (PLEG): container finished" podID="75845278-3169-49ce-ace9-40a8a6ee001a" containerID="8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968" exitCode=0
Nov 02 15:37:44.946492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.946289    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"
Nov 02 15:37:44.946492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.946338    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" event={"ID":"75845278-3169-49ce-ace9-40a8a6ee001a","Type":"ContainerDied","Data":"8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968"}
Nov 02 15:37:44.946492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.946381    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" event={"ID":"75845278-3169-49ce-ace9-40a8a6ee001a","Type":"ContainerDied","Data":"ec0d85ea06d0e562bde6dd5645653ca8cbb09c612c4b312a0b4dd3c392358d55"}
Nov 02 15:37:44.946492 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.946402    4891 scope.go:122] "RemoveContainer" containerID="8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968"
Nov 02 15:37:44.947947 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.947920    4891 generic.go:356] "Generic (PLEG): container finished" podID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerID="c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8" exitCode=0
Nov 02 15:37:44.948068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.947975    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"
Nov 02 15:37:44.948068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.948009    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" event={"ID":"2dbba973-9a21-4bed-9112-e9d0f9282d68","Type":"ContainerDied","Data":"c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8"}
Nov 02 15:37:44.948068 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.948055    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c" event={"ID":"2dbba973-9a21-4bed-9112-e9d0f9282d68","Type":"ContainerDied","Data":"39ce053f951f990510ea29351abd272375d578fd26020905bd0df37f5ae884b4"}
Nov 02 15:37:44.949582 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.949562    4891 generic.go:356] "Generic (PLEG): container finished" podID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerID="5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7" exitCode=0
Nov 02 15:37:44.949683 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.949646    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" event={"ID":"73d2092c-52f2-499d-85f7-8681595ea6b3","Type":"ContainerDied","Data":"5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7"}
Nov 02 15:37:44.949683 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.949664    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"
Nov 02 15:37:44.949770 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.949668    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" event={"ID":"73d2092c-52f2-499d-85f7-8681595ea6b3","Type":"ContainerDied","Data":"5051fcde78f219c549c759121e99746cd25923906e7d4b0979bcd0d9949b2b61"}
Nov 02 15:37:44.958379 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.958179    4891 scope.go:122] "RemoveContainer" containerID="8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968"
Nov 02 15:37:44.958587 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:37:44.958557    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968\": container with ID starting with 8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968 not found: ID does not exist" containerID="8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968"
Nov 02 15:37:44.958654 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.958597    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968"} err="failed to get container status \"8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968\": rpc error: code = NotFound desc = could not find container \"8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968\": container with ID starting with 8c107907845dd14bc38e11f9ee4786b5aad23d6517d44519623e5d97e755f968 not found: ID does not exist"
Nov 02 15:37:44.958654 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.958619    4891 scope.go:122] "RemoveContainer" containerID="c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8"
Nov 02 15:37:44.971447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.968628    4891 scope.go:122] "RemoveContainer" containerID="c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8"
Nov 02 15:37:44.971447 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:37:44.971409    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8\": container with ID starting with c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8 not found: ID does not exist" containerID="c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8"
Nov 02 15:37:44.971447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.971432    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8"} err="failed to get container status \"c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8\": rpc error: code = NotFound desc = could not find container \"c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8\": container with ID starting with c28488d56a202110f69520edd07aa72ce141a169e971c5e4d0195e94bf5ef6f8 not found: ID does not exist"
Nov 02 15:37:44.971634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.971450    4891 scope.go:122] "RemoveContainer" containerID="5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7"
Nov 02 15:37:44.982433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.981428    4891 scope.go:122] "RemoveContainer" containerID="5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7"
Nov 02 15:37:44.983101 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:37:44.983073    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7\": container with ID starting with 5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7 not found: ID does not exist" containerID="5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7"
Nov 02 15:37:44.983207 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:44.983114    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7"} err="failed to get container status \"5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7\": rpc error: code = NotFound desc = could not find container \"5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7\": container with ID starting with 5d8ca614a5636fc24fd7a6fcf558f1fabb3eb34f2b63538404300fecf4b554d7 not found: ID does not exist"
Nov 02 15:37:45.114961 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:45.114927    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"]
Nov 02 15:37:45.126149 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:45.122967    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs"]
Nov 02 15:37:45.230191 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:45.230160    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"]
Nov 02 15:37:45.234757 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:45.234731    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b"]
Nov 02 15:37:45.343255 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:45.343221    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"]
Nov 02 15:37:45.349439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:45.349409    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c18e79/simulator-w6-c18e79-kserve-6997cbd7fb-fvm4c"]
Nov 02 15:37:45.506520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:45.506413    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/distinct-w6-c18e79-kserve-5546557dcb-6hg5b" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:37:45.506670 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:45.506431    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c18e79/premium-w6-c18e79-kserve-64f6f85cbf-gzfvs" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:37:46.522706 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:46.522674    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" path="/var/lib/kubelet/pods/2dbba973-9a21-4bed-9112-e9d0f9282d68/volumes"
Nov 02 15:37:46.523137 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:46.523123    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" path="/var/lib/kubelet/pods/73d2092c-52f2-499d-85f7-8681595ea6b3/volumes"
Nov 02 15:37:46.523547 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:46.523534    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" path="/var/lib/kubelet/pods/75845278-3169-49ce-ace9-40a8a6ee001a/volumes"
Nov 02 15:37:47.303979 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:47.303938    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26" podUID="08e8a676-324b-450f-85b2-8ec2df940dae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:53.052376 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:53.052305    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"]
Nov 02 15:37:53.102094 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:53.102031    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fdc8d7-8nmcr"]
Nov 02 15:37:54.521948 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.521919    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab649d1a-82d5-499c-be05-2e0ce9d8c968" path="/var/lib/kubelet/pods/ab649d1a-82d5-499c-be05-2e0ce9d8c968/volumes"
Nov 02 15:37:54.769023 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.768987    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"]
Nov 02 15:37:54.769509 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769494    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44" containerName="revoke-keys"
Nov 02 15:37:54.769569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769511    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44" containerName="revoke-keys"
Nov 02 15:37:54.769569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769528    4891 cpu_manager.go:414] "containerMap: removing container" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main"
Nov 02 15:37:54.769569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769534    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main"
Nov 02 15:37:54.769569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769544    4891 cpu_manager.go:414] "containerMap: removing container" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main"
Nov 02 15:37:54.769569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769550    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main"
Nov 02 15:37:54.769569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769562    4891 cpu_manager.go:414] "containerMap: removing container" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main"
Nov 02 15:37:54.769569 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769567    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main"
Nov 02 15:37:54.769781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769583    4891 cpu_manager.go:414] "containerMap: removing container" podUID="7304fe30-39ac-4e9a-b880-20cc50287018" containerName="istio-proxy"
Nov 02 15:37:54.769781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769588    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7304fe30-39ac-4e9a-b880-20cc50287018" containerName="istio-proxy"
Nov 02 15:37:54.769781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769673    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="7304fe30-39ac-4e9a-b880-20cc50287018" containerName="istio-proxy"
Nov 02 15:37:54.769781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769684    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44" containerName="revoke-keys"
Nov 02 15:37:54.769781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769694    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="75845278-3169-49ce-ace9-40a8a6ee001a" containerName="main"
Nov 02 15:37:54.769781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769703    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dbba973-9a21-4bed-9112-e9d0f9282d68" containerName="main"
Nov 02 15:37:54.769781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.769713    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d2092c-52f2-499d-85f7-8681595ea6b3" containerName="main"
Nov 02 15:37:54.774707 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.774665    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:54.830913 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.830882    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"]
Nov 02 15:37:54.876162 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.876131    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:37:54.876389 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.876169    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:37:54.942631 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.942598    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c064032d-bflnc\" (UID: \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:54.942799 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.942664    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c064032d-bflnc\" (UID: \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:54.942838 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:54.942786    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92q8q\" (UniqueName: \"kubernetes.io/projected/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-kube-api-access-92q8q\") pod \"maas-api-revoke-sub-c064032d-bflnc\" (UID: \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:55.044214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:55.044125    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c064032d-bflnc\" (UID: \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:55.044214 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:55.044178    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c064032d-bflnc\" (UID: \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:55.044511 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:55.044221    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92q8q\" (UniqueName: \"kubernetes.io/projected/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-kube-api-access-92q8q\") pod \"maas-api-revoke-sub-c064032d-bflnc\" (UID: \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:55.044873 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:55.044850    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c064032d-bflnc\" (UID: \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:55.046632 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:55.046611    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c064032d-bflnc\" (UID: \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:55.104924 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:55.104891    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92q8q\" (UniqueName: \"kubernetes.io/projected/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-kube-api-access-92q8q\") pod \"maas-api-revoke-sub-c064032d-bflnc\" (UID: \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:55.388605 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:55.388520    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:55.769495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:55.769466    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"]
Nov 02 15:37:55.770573 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:37:55.770539    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edbba32_9cdc_4bcb_ae47_aabfb8278f6d.slice/crio-75e2652147eb1eb60cda07617045cbbb33f50bf3d4dbd98e8df010133d9085a8 WatchSource:0}: Error finding container 75e2652147eb1eb60cda07617045cbbb33f50bf3d4dbd98e8df010133d9085a8: Status 404 returned error can't find the container with id 75e2652147eb1eb60cda07617045cbbb33f50bf3d4dbd98e8df010133d9085a8
Nov 02 15:37:56.046791 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:56.046756    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc" event={"ID":"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d","Type":"ContainerStarted","Data":"75e2652147eb1eb60cda07617045cbbb33f50bf3d4dbd98e8df010133d9085a8"}
Nov 02 15:37:57.054834 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:57.054794    4891 generic.go:356] "Generic (PLEG): container finished" podID="7edbba32-9cdc-4bcb-ae47-aabfb8278f6d" containerID="7351e6bcd4c28bdd2d09e82eed2aa4d9f2abc639c539d26b6a90d969fb35d0fe" exitCode=0
Nov 02 15:37:57.055240 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:57.054881    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc" event={"ID":"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d","Type":"ContainerDied","Data":"7351e6bcd4c28bdd2d09e82eed2aa4d9f2abc639c539d26b6a90d969fb35d0fe"}
Nov 02 15:37:57.303834 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:57.303793    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26" podUID="08e8a676-324b-450f-85b2-8ec2df940dae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:58.193036 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:58.193009    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:58.378474 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:58.378391    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-openshift-ca\") pod \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\" (UID: \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\") "
Nov 02 15:37:58.378474 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:58.378440    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-kube-api-access-92q8q\" (UniqueName: \"kubernetes.io/projected/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-kube-api-access-92q8q\") pod \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\" (UID: \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\") "
Nov 02 15:37:58.378673 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:58.378529    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-xks-ca\") pod \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\" (UID: \"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d\") "
Nov 02 15:37:58.378808 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:58.378781    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-openshift-ca" pod "7edbba32-9cdc-4bcb-ae47-aabfb8278f6d" (UID: "7edbba32-9cdc-4bcb-ae47-aabfb8278f6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:58.380720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:58.380692    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-xks-ca" pod "7edbba32-9cdc-4bcb-ae47-aabfb8278f6d" (UID: "7edbba32-9cdc-4bcb-ae47-aabfb8278f6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:58.380720 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:58.380710    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-kube-api-access-92q8q" pod "7edbba32-9cdc-4bcb-ae47-aabfb8278f6d" (UID: "7edbba32-9cdc-4bcb-ae47-aabfb8278f6d"). InnerVolumeSpecName "kube-api-access-92q8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:58.479879 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:58.479844    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92q8q\" (UniqueName: \"kubernetes.io/projected/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-kube-api-access-92q8q\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:58.479879 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:58.479872    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:58.479879 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:58.479883    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:37:59.068733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:59.068701    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"
Nov 02 15:37:59.068907 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:59.068704    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc" event={"ID":"7edbba32-9cdc-4bcb-ae47-aabfb8278f6d","Type":"ContainerDied","Data":"75e2652147eb1eb60cda07617045cbbb33f50bf3d4dbd98e8df010133d9085a8"}
Nov 02 15:37:59.068907 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:37:59.068823    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75e2652147eb1eb60cda07617045cbbb33f50bf3d4dbd98e8df010133d9085a8"
Nov 02 15:38:01.582534 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:01.582508    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"
Nov 02 15:38:01.709560 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:01.709486    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e8a676-324b-450f-85b2-8ec2df940dae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e8a676-324b-450f-85b2-8ec2df940dae-plugins-config-volume\") pod \"08e8a676-324b-450f-85b2-8ec2df940dae\" (UID: \"08e8a676-324b-450f-85b2-8ec2df940dae\") "
Nov 02 15:38:01.709702 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:01.709643    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e8a676-324b-450f-85b2-8ec2df940dae-kube-api-access-86p4v\" (UniqueName: \"kubernetes.io/projected/08e8a676-324b-450f-85b2-8ec2df940dae-kube-api-access-86p4v\") pod \"08e8a676-324b-450f-85b2-8ec2df940dae\" (UID: \"08e8a676-324b-450f-85b2-8ec2df940dae\") "
Nov 02 15:38:01.709927 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:01.709898    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e8a676-324b-450f-85b2-8ec2df940dae-plugins-config-volume" pod "08e8a676-324b-450f-85b2-8ec2df940dae" (UID: "08e8a676-324b-450f-85b2-8ec2df940dae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:38:01.711754 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:01.711726    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e8a676-324b-450f-85b2-8ec2df940dae-kube-api-access-86p4v" pod "08e8a676-324b-450f-85b2-8ec2df940dae" (UID: "08e8a676-324b-450f-85b2-8ec2df940dae"). InnerVolumeSpecName "kube-api-access-86p4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:38:01.810867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:01.810827    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86p4v\" (UniqueName: \"kubernetes.io/projected/08e8a676-324b-450f-85b2-8ec2df940dae-kube-api-access-86p4v\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:38:01.810867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:01.810861    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e8a676-324b-450f-85b2-8ec2df940dae-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:38:02.089342 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:02.089288    4891 generic.go:356] "Generic (PLEG): container finished" podID="08e8a676-324b-450f-85b2-8ec2df940dae" containerID="158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88" exitCode=0
Nov 02 15:38:02.089506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:02.089343    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"
Nov 02 15:38:02.089506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:02.089371    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26" event={"ID":"08e8a676-324b-450f-85b2-8ec2df940dae","Type":"ContainerDied","Data":"158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88"}
Nov 02 15:38:02.089506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:02.089430    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26" event={"ID":"08e8a676-324b-450f-85b2-8ec2df940dae","Type":"ContainerDied","Data":"fe3fb7fa91734949c0b7480b4e6ccd0d739ca27771d3bacb779e48cd6db35ab0"}
Nov 02 15:38:02.089506 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:02.089460    4891 scope.go:122] "RemoveContainer" containerID="158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88"
Nov 02 15:38:02.104538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:02.104514    4891 scope.go:122] "RemoveContainer" containerID="158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88"
Nov 02 15:38:02.104878 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:38:02.104858    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88\": container with ID starting with 158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88 not found: ID does not exist" containerID="158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88"
Nov 02 15:38:02.104952 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:02.104886    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88"} err="failed to get container status \"158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88\": rpc error: code = NotFound desc = could not find container \"158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88\": container with ID starting with 158797dab5dfeae5cac046a7f1f824c018035745094083372402d7988dbafc88 not found: ID does not exist"
Nov 02 15:38:02.256255 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:02.256145    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"]
Nov 02 15:38:02.273832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:02.264086    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-c18e79-5fcfd69bc9-dmk26"]
Nov 02 15:38:02.522259 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:02.522217    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e8a676-324b-450f-85b2-8ec2df940dae" path="/var/lib/kubelet/pods/08e8a676-324b-450f-85b2-8ec2df940dae/volumes"
Nov 02 15:38:14.274465 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:38:14.274426    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:38:31.545014 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:31.544971    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"]
Nov 02 15:38:31.594495 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:31.594457    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c18e79-5fv8g"]
Nov 02 15:38:32.521679 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:32.521649    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44" path="/var/lib/kubelet/pods/f2c2c23b-a1d8-4ce0-b962-92d1aaa43c44/volumes"
Nov 02 15:38:35.560812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:35.560782    4891 scope.go:122] "RemoveContainer" containerID="1318caefb90420c6948b2a7b05abf395d4bb61c80c366d57dbb61543e93acd5b"
Nov 02 15:38:35.570354 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:35.570296    4891 scope.go:122] "RemoveContainer" containerID="8f675769c6db7102ef7811d46d91212a0c42cbe4f8ef7d23d66d81ee9d682e90"
Nov 02 15:38:35.579438 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:35.579411    4891 scope.go:122] "RemoveContainer" containerID="e35cefaf59a65c078ba122c6d8ea445cecdc9fbd20defe2a8e3338a0d50c7524"
Nov 02 15:38:39.270849 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:39.270815    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"]
Nov 02 15:38:39.271371 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:39.271075    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" podUID="c325c48e-6940-4af7-b486-488df4754813" containerName="istio-proxy" containerID="cri-o://f321cd99b7e07d9671844b0a7eafc9242c5858f88bc57493a02d3c1300e5d6cc" gracePeriod=30
Nov 02 15:38:44.273277 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:38:44.273239    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:38:44.391095 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.390768    4891 generic.go:356] "Generic (PLEG): container finished" podID="c325c48e-6940-4af7-b486-488df4754813" containerID="f321cd99b7e07d9671844b0a7eafc9242c5858f88bc57493a02d3c1300e5d6cc" exitCode=0
Nov 02 15:38:44.391095 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.390828    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" event={"ID":"c325c48e-6940-4af7-b486-488df4754813","Type":"ContainerDied","Data":"f321cd99b7e07d9671844b0a7eafc9242c5858f88bc57493a02d3c1300e5d6cc"}
Nov 02 15:38:44.526120 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.526054    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:38:44.584952 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.584919    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-istio-token\" (UniqueName: \"kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-istio-token\") pod \"c325c48e-6940-4af7-b486-488df4754813\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") "
Nov 02 15:38:44.585172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.584975    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c325c48e-6940-4af7-b486-488df4754813-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c325c48e-6940-4af7-b486-488df4754813-istio-podinfo\") pod \"c325c48e-6940-4af7-b486-488df4754813\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") "
Nov 02 15:38:44.585172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585039    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c325c48e-6940-4af7-b486-488df4754813-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c325c48e-6940-4af7-b486-488df4754813-istiod-ca-cert\") pod \"c325c48e-6940-4af7-b486-488df4754813\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") "
Nov 02 15:38:44.585172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585065    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-envoy\") pod \"c325c48e-6940-4af7-b486-488df4754813\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") "
Nov 02 15:38:44.585172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585105    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-kube-api-access-kwlfz\" (UniqueName: \"kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-kube-api-access-kwlfz\") pod \"c325c48e-6940-4af7-b486-488df4754813\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") "
Nov 02 15:38:44.585172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585126    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-certs\") pod \"c325c48e-6940-4af7-b486-488df4754813\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") "
Nov 02 15:38:44.585172 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585156    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-data\") pod \"c325c48e-6940-4af7-b486-488df4754813\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") "
Nov 02 15:38:44.585574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585203    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-credential-socket\") pod \"c325c48e-6940-4af7-b486-488df4754813\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") "
Nov 02 15:38:44.585574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585232    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-socket\") pod \"c325c48e-6940-4af7-b486-488df4754813\" (UID: \"c325c48e-6940-4af7-b486-488df4754813\") "
Nov 02 15:38:44.585574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585465    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-certs" pod "c325c48e-6940-4af7-b486-488df4754813" (UID: "c325c48e-6940-4af7-b486-488df4754813"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:44.585574 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585516    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-data" pod "c325c48e-6940-4af7-b486-488df4754813" (UID: "c325c48e-6940-4af7-b486-488df4754813"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:44.585797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585550    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c325c48e-6940-4af7-b486-488df4754813-istiod-ca-cert" pod "c325c48e-6940-4af7-b486-488df4754813" (UID: "c325c48e-6940-4af7-b486-488df4754813"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:38:44.585797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585629    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-credential-socket" pod "c325c48e-6940-4af7-b486-488df4754813" (UID: "c325c48e-6940-4af7-b486-488df4754813"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:44.585797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585647    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-socket" pod "c325c48e-6940-4af7-b486-488df4754813" (UID: "c325c48e-6940-4af7-b486-488df4754813"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:44.585797 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585783    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c325c48e-6940-4af7-b486-488df4754813-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:38:44.586022 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585807    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:38:44.586022 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585823    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:38:44.586022 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585835    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:38:44.586022 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.585851    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:38:44.587414 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.587386    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-istio-token" pod "c325c48e-6940-4af7-b486-488df4754813" (UID: "c325c48e-6940-4af7-b486-488df4754813"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:38:44.587545 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.587443    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-kube-api-access-kwlfz" pod "c325c48e-6940-4af7-b486-488df4754813" (UID: "c325c48e-6940-4af7-b486-488df4754813"). InnerVolumeSpecName "kube-api-access-kwlfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:38:44.587592 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.587546    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-envoy" pod "c325c48e-6940-4af7-b486-488df4754813" (UID: "c325c48e-6940-4af7-b486-488df4754813"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:44.587633 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.587610    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c325c48e-6940-4af7-b486-488df4754813-istio-podinfo" pod "c325c48e-6940-4af7-b486-488df4754813" (UID: "c325c48e-6940-4af7-b486-488df4754813"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:38:44.686917 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.686887    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:38:44.686917 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.686916    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c325c48e-6940-4af7-b486-488df4754813-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:38:44.686917 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.686927    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c325c48e-6940-4af7-b486-488df4754813-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:38:44.687156 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:44.686941    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwlfz\" (UniqueName: \"kubernetes.io/projected/c325c48e-6940-4af7-b486-488df4754813-kube-api-access-kwlfz\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:38:45.398699 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:45.398670    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"
Nov 02 15:38:45.399168 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:45.398701    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5" event={"ID":"c325c48e-6940-4af7-b486-488df4754813","Type":"ContainerDied","Data":"14ac8edb72caae06a5073ea79d8b4e8c0d4f7b95146279471ac9204ae9fe7102"}
Nov 02 15:38:45.399168 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:45.398763    4891 scope.go:122] "RemoveContainer" containerID="f321cd99b7e07d9671844b0a7eafc9242c5858f88bc57493a02d3c1300e5d6cc"
Nov 02 15:38:45.581816 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:45.581771    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"]
Nov 02 15:38:45.629976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:45.629942    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c18e79-openshift-default-59544557d7-b9xj5"]
Nov 02 15:38:46.520922 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:38:46.520888    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c325c48e-6940-4af7-b486-488df4754813" path="/var/lib/kubelet/pods/c325c48e-6940-4af7-b486-488df4754813/volumes"
Nov 02 15:39:09.865498 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:09.865463    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"]
Nov 02 15:39:14.273525 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:39:14.273445    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:39:24.885650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:24.885615    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg" podUID="5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca" containerName="main" containerID="cri-o://58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2" gracePeriod=45
Nov 02 15:39:25.145129 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.145065    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:39:25.197386 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.197347    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-model-cache\") pod \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") "
Nov 02 15:39:25.197581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.197393    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tmp-dir\") pod \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") "
Nov 02 15:39:25.197581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.197420    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-kube-api-access-btdvz\" (UniqueName: \"kubernetes.io/projected/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-kube-api-access-btdvz\") pod \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") "
Nov 02 15:39:25.197581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.197502    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tls-certs\") pod \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") "
Nov 02 15:39:25.197581 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.197565    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-dshm\") pod \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") "
Nov 02 15:39:25.197829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.197613    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-home\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-home\") pod \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\" (UID: \"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca\") "
Nov 02 15:39:25.197829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.197651    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-model-cache" pod "5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca" (UID: "5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:25.197829 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.197719    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tmp-dir" pod "5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca" (UID: "5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:25.197997 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.197927    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-home" pod "5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca" (UID: "5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:25.198053 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.198041    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:39:25.198104 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.198059    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:39:25.198104 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.198073    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:39:25.199832 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.199805    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-kube-api-access-btdvz" pod "5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca" (UID: "5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca"). InnerVolumeSpecName "kube-api-access-btdvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:39:25.199951 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.199828    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tls-certs" pod "5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca" (UID: "5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:39:25.199951 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.199848    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-dshm" pod "5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca" (UID: "5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:25.299684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.299646    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:39:25.299684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.299679    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:39:25.299684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.299688    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btdvz\" (UniqueName: \"kubernetes.io/projected/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca-kube-api-access-btdvz\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:39:25.657094 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.657058    4891 generic.go:356] "Generic (PLEG): container finished" podID="5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca" containerID="58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2" exitCode=0
Nov 02 15:39:25.657272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.657099    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"
Nov 02 15:39:25.657272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.657140    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg" event={"ID":"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca","Type":"ContainerDied","Data":"58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2"}
Nov 02 15:39:25.657272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.657187    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg" event={"ID":"5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca","Type":"ContainerDied","Data":"aad96aa0bfbee42b288d07d42d1a81f8d8387af0c512779d460e785b5db536c4"}
Nov 02 15:39:25.657272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.657210    4891 scope.go:122] "RemoveContainer" containerID="58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2"
Nov 02 15:39:25.667685 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.667663    4891 scope.go:122] "RemoveContainer" containerID="58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2"
Nov 02 15:39:25.667989 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:39:25.667968    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2\": container with ID starting with 58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2 not found: ID does not exist" containerID="58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2"
Nov 02 15:39:25.668034 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.667998    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2"} err="failed to get container status \"58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2\": rpc error: code = NotFound desc = could not find container \"58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2\": container with ID starting with 58836ca6304540e9c0d3910c68d3bae37bdf7eb5872d83e93c42d99062380fc2 not found: ID does not exist"
Nov 02 15:39:25.829439 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.829403    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"]
Nov 02 15:39:25.834659 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:25.834631    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-39098c/test-model-39098c-kserve-5d7b4dd687-pnftg"]
Nov 02 15:39:26.521487 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:26.521458    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca" path="/var/lib/kubelet/pods/5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca/volumes"
Nov 02 15:39:35.633725 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:35.633692    4891 scope.go:122] "RemoveContainer" containerID="fa5e707abedea2afa1d45cc285c08b6236988fb516f25b24936bd043c397bb62"
Nov 02 15:39:35.643521 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:35.643494    4891 scope.go:122] "RemoveContainer" containerID="d7be6d119ef8ff58d1f211b0f5040e213dab3637c7ec410a9ae0b8e5290c9e21"
Nov 02 15:39:44.274497 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:39:44.274458    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:39:46.582626 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:46.582592    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"]
Nov 02 15:39:46.692084 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:46.692051    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"]
Nov 02 15:39:51.583433 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:51.583373    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26" podUID="e8e79f6e-44a4-4f84-a651-8963c8299b39" containerName="payload-processing" containerID="cri-o://f7d58a6b3ef575c9ef5630f2460b101f9d7a544ed357599d8b8fc3394fcc68a7" gracePeriod=25
Nov 02 15:39:51.693189 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:51.693115    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x" podUID="cd686938-df40-4efe-ad36-53c6337f78e4" containerName="payload-pre-processing" containerID="cri-o://4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff" gracePeriod=25
Nov 02 15:39:51.823670 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:51.823634    4891 generic.go:356] "Generic (PLEG): container finished" podID="e8e79f6e-44a4-4f84-a651-8963c8299b39" containerID="f7d58a6b3ef575c9ef5630f2460b101f9d7a544ed357599d8b8fc3394fcc68a7" exitCode=0
Nov 02 15:39:51.823804 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:51.823706    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26" event={"ID":"e8e79f6e-44a4-4f84-a651-8963c8299b39","Type":"ContainerDied","Data":"f7d58a6b3ef575c9ef5630f2460b101f9d7a544ed357599d8b8fc3394fcc68a7"}
Nov 02 15:39:51.843260 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:51.843206    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"
Nov 02 15:39:51.867464 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:51.867433    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e79f6e-44a4-4f84-a651-8963c8299b39-kube-api-access-drmkx\" (UniqueName: \"kubernetes.io/projected/e8e79f6e-44a4-4f84-a651-8963c8299b39-kube-api-access-drmkx\") pod \"e8e79f6e-44a4-4f84-a651-8963c8299b39\" (UID: \"e8e79f6e-44a4-4f84-a651-8963c8299b39\") "
Nov 02 15:39:51.867614 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:51.867505    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e79f6e-44a4-4f84-a651-8963c8299b39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e79f6e-44a4-4f84-a651-8963c8299b39-plugins-config-volume\") pod \"e8e79f6e-44a4-4f84-a651-8963c8299b39\" (UID: \"e8e79f6e-44a4-4f84-a651-8963c8299b39\") "
Nov 02 15:39:51.867945 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:51.867920    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e79f6e-44a4-4f84-a651-8963c8299b39-plugins-config-volume" pod "e8e79f6e-44a4-4f84-a651-8963c8299b39" (UID: "e8e79f6e-44a4-4f84-a651-8963c8299b39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:39:51.869710 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:51.869687    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e79f6e-44a4-4f84-a651-8963c8299b39-kube-api-access-drmkx" pod "e8e79f6e-44a4-4f84-a651-8963c8299b39" (UID: "e8e79f6e-44a4-4f84-a651-8963c8299b39"). InnerVolumeSpecName "kube-api-access-drmkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:39:51.968258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:51.968227    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e79f6e-44a4-4f84-a651-8963c8299b39-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:39:51.968258 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:51.968255    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drmkx\" (UniqueName: \"kubernetes.io/projected/e8e79f6e-44a4-4f84-a651-8963c8299b39-kube-api-access-drmkx\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:39:52.061903 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:52.061869    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"]
Nov 02 15:39:52.065742 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:52.065712    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515aa7f2-m65s5"]
Nov 02 15:39:52.521579 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:52.521550    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a836680-5c62-4207-861f-cf9b8875ae11" path="/var/lib/kubelet/pods/4a836680-5c62-4207-861f-cf9b8875ae11/volumes"
Nov 02 15:39:52.830965 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:52.830874    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"
Nov 02 15:39:52.830965 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:52.830910    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26" event={"ID":"e8e79f6e-44a4-4f84-a651-8963c8299b39","Type":"ContainerDied","Data":"59b7f547672466cda1b038bd417ece0447f3370de4b00699875a1734a4fd71de"}
Nov 02 15:39:52.831505 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:52.830974    4891 scope.go:122] "RemoveContainer" containerID="f7d58a6b3ef575c9ef5630f2460b101f9d7a544ed357599d8b8fc3394fcc68a7"
Nov 02 15:39:52.994625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:52.994593    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"]
Nov 02 15:39:53.044037 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:53.044001    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-39098c-7dfd88f45c-5mh26"]
Nov 02 15:39:53.995565 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:53.995535    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"]
Nov 02 15:39:53.995983 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:53.995726    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn" podUID="f0a43216-2f54-4378-82c5-7081f1bec36f" containerName="maas-api" containerID="cri-o://c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052" gracePeriod=30
Nov 02 15:39:54.268186 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.268118    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:39:54.289064 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.289037    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a43216-2f54-4378-82c5-7081f1bec36f-kube-api-access-4cjkg\" (UniqueName: \"kubernetes.io/projected/f0a43216-2f54-4378-82c5-7081f1bec36f-kube-api-access-4cjkg\") pod \"f0a43216-2f54-4378-82c5-7081f1bec36f\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") "
Nov 02 15:39:54.289234 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.289086    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a43216-2f54-4378-82c5-7081f1bec36f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a43216-2f54-4378-82c5-7081f1bec36f-openshift-service-ca\") pod \"f0a43216-2f54-4378-82c5-7081f1bec36f\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") "
Nov 02 15:39:54.289234 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.289175    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-metrics-certs\") pod \"f0a43216-2f54-4378-82c5-7081f1bec36f\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") "
Nov 02 15:39:54.289234 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.289222    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-maas-api-tls\") pod \"f0a43216-2f54-4378-82c5-7081f1bec36f\" (UID: \"f0a43216-2f54-4378-82c5-7081f1bec36f\") "
Nov 02 15:39:54.289538 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.289508    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a43216-2f54-4378-82c5-7081f1bec36f-openshift-service-ca" pod "f0a43216-2f54-4378-82c5-7081f1bec36f" (UID: "f0a43216-2f54-4378-82c5-7081f1bec36f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:39:54.291280 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.291250    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-maas-api-tls" pod "f0a43216-2f54-4378-82c5-7081f1bec36f" (UID: "f0a43216-2f54-4378-82c5-7081f1bec36f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:39:54.291404 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.291324    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a43216-2f54-4378-82c5-7081f1bec36f-kube-api-access-4cjkg" pod "f0a43216-2f54-4378-82c5-7081f1bec36f" (UID: "f0a43216-2f54-4378-82c5-7081f1bec36f"). InnerVolumeSpecName "kube-api-access-4cjkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:39:54.291404 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.291370    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-metrics-certs" pod "f0a43216-2f54-4378-82c5-7081f1bec36f" (UID: "f0a43216-2f54-4378-82c5-7081f1bec36f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:39:54.403344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.396066    4891 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a43216-2f54-4378-82c5-7081f1bec36f-openshift-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:39:54.403344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.396106    4891 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-metrics-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:39:54.403344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.396120    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0a43216-2f54-4378-82c5-7081f1bec36f-maas-api-tls\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:39:54.403344 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.396134    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cjkg\" (UniqueName: \"kubernetes.io/projected/f0a43216-2f54-4378-82c5-7081f1bec36f-kube-api-access-4cjkg\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:39:54.522825 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.522791    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e79f6e-44a4-4f84-a651-8963c8299b39" path="/var/lib/kubelet/pods/e8e79f6e-44a4-4f84-a651-8963c8299b39/volumes"
Nov 02 15:39:54.845128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.845041    4891 generic.go:356] "Generic (PLEG): container finished" podID="f0a43216-2f54-4378-82c5-7081f1bec36f" containerID="c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052" exitCode=0
Nov 02 15:39:54.845128 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.845087    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"
Nov 02 15:39:54.845347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.845133    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn" event={"ID":"f0a43216-2f54-4378-82c5-7081f1bec36f","Type":"ContainerDied","Data":"c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052"}
Nov 02 15:39:54.845347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.845184    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn" event={"ID":"f0a43216-2f54-4378-82c5-7081f1bec36f","Type":"ContainerDied","Data":"7ad87ada4b0be0760e90c967359f75313f7d144a9e9c7163b20063f1b7ec706e"}
Nov 02 15:39:54.845347 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.845211    4891 scope.go:122] "RemoveContainer" containerID="c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052"
Nov 02 15:39:54.854662 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.854645    4891 scope.go:122] "RemoveContainer" containerID="c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052"
Nov 02 15:39:54.854939 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:39:54.854921    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052\": container with ID starting with c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052 not found: ID does not exist" containerID="c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052"
Nov 02 15:39:54.854993 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.854951    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052"} err="failed to get container status \"c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052\": rpc error: code = NotFound desc = could not find container \"c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052\": container with ID starting with c57bf197b938ca0d07ed94b7c380d6a676592fcb0ba54816577c102f97a3f052 not found: ID does not exist"
Nov 02 15:39:54.871295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:54.871263    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x" podUID="cd686938-df40-4efe-ad36-53c6337f78e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:39:55.012783 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:55.012753    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"]
Nov 02 15:39:55.020643 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:55.020612    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-39098c-67d6db6db-cx4tn"]
Nov 02 15:39:56.521650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:39:56.521620    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a43216-2f54-4378-82c5-7081f1bec36f" path="/var/lib/kubelet/pods/f0a43216-2f54-4378-82c5-7081f1bec36f/volumes"
Nov 02 15:40:03.927555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:03.927522    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"]
Nov 02 15:40:03.928021 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:03.927732    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy" containerID="cri-o://7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e" gracePeriod=30
Nov 02 15:40:04.807691 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.807652    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"]
Nov 02 15:40:04.808095 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808082    4891 cpu_manager.go:414] "containerMap: removing container" podUID="08e8a676-324b-450f-85b2-8ec2df940dae" containerName="payload-pre-processing"
Nov 02 15:40:04.808145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808098    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e8a676-324b-450f-85b2-8ec2df940dae" containerName="payload-pre-processing"
Nov 02 15:40:04.808145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808110    4891 cpu_manager.go:414] "containerMap: removing container" podUID="e8e79f6e-44a4-4f84-a651-8963c8299b39" containerName="payload-processing"
Nov 02 15:40:04.808145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808115    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e79f6e-44a4-4f84-a651-8963c8299b39" containerName="payload-processing"
Nov 02 15:40:04.808145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808138    4891 cpu_manager.go:414] "containerMap: removing container" podUID="c325c48e-6940-4af7-b486-488df4754813" containerName="istio-proxy"
Nov 02 15:40:04.808145 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808143    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c325c48e-6940-4af7-b486-488df4754813" containerName="istio-proxy"
Nov 02 15:40:04.808304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808152    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f0a43216-2f54-4378-82c5-7081f1bec36f" containerName="maas-api"
Nov 02 15:40:04.808304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808156    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a43216-2f54-4378-82c5-7081f1bec36f" containerName="maas-api"
Nov 02 15:40:04.808304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808164    4891 cpu_manager.go:414] "containerMap: removing container" podUID="5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca" containerName="main"
Nov 02 15:40:04.808304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808168    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca" containerName="main"
Nov 02 15:40:04.808304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808176    4891 cpu_manager.go:414] "containerMap: removing container" podUID="7edbba32-9cdc-4bcb-ae47-aabfb8278f6d" containerName="revoke-keys"
Nov 02 15:40:04.808304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808180    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7edbba32-9cdc-4bcb-ae47-aabfb8278f6d" containerName="revoke-keys"
Nov 02 15:40:04.808304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808245    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e8a676-324b-450f-85b2-8ec2df940dae" containerName="payload-pre-processing"
Nov 02 15:40:04.808304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808254    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e79f6e-44a4-4f84-a651-8963c8299b39" containerName="payload-processing"
Nov 02 15:40:04.808304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808263    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="c325c48e-6940-4af7-b486-488df4754813" containerName="istio-proxy"
Nov 02 15:40:04.808304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808270    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a43216-2f54-4378-82c5-7081f1bec36f" containerName="maas-api"
Nov 02 15:40:04.808304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808276    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="5419e5ef-a3cb-4a24-a8b8-fd0e9da5d7ca" containerName="main"
Nov 02 15:40:04.808304 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.808281    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="7edbba32-9cdc-4bcb-ae47-aabfb8278f6d" containerName="revoke-keys"
Nov 02 15:40:04.811400 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.811384    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:04.871208 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.871166    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x" podUID="cd686938-df40-4efe-ad36-53c6337f78e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:40:04.916156 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.916086    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:40:04.916395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.916159    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:40:04.922457 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.922418    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"]
Nov 02 15:40:04.926101 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.926073    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz\" (UID: \"a492f8b8-8322-402f-8803-fcd6735e897e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:04.926229 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.926152    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz\" (UID: \"a492f8b8-8322-402f-8803-fcd6735e897e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:04.926229 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.926187    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlcn8\" (UniqueName: \"kubernetes.io/projected/a492f8b8-8322-402f-8803-fcd6735e897e-kube-api-access-mlcn8\") pod \"maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz\" (UID: \"a492f8b8-8322-402f-8803-fcd6735e897e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:04.965416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.965381    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:40:04.965800 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:04.965471    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:40:05.027784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:05.027742    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz\" (UID: \"a492f8b8-8322-402f-8803-fcd6735e897e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:05.027994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:05.027793    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlcn8\" (UniqueName: \"kubernetes.io/projected/a492f8b8-8322-402f-8803-fcd6735e897e-kube-api-access-mlcn8\") pod \"maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz\" (UID: \"a492f8b8-8322-402f-8803-fcd6735e897e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:05.027994 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:05.027953    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz\" (UID: \"a492f8b8-8322-402f-8803-fcd6735e897e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:05.028634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:05.028616    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz\" (UID: \"a492f8b8-8322-402f-8803-fcd6735e897e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:05.030179 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:05.030161    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz\" (UID: \"a492f8b8-8322-402f-8803-fcd6735e897e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:05.093086 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:05.093011    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlcn8\" (UniqueName: \"kubernetes.io/projected/a492f8b8-8322-402f-8803-fcd6735e897e-kube-api-access-mlcn8\") pod \"maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz\" (UID: \"a492f8b8-8322-402f-8803-fcd6735e897e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:05.124411 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:05.124376    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:05.503244 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:05.503221    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"]
Nov 02 15:40:05.505556 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:40:05.505521    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda492f8b8_8322_402f_8803_fcd6735e897e.slice/crio-721ff63903720cbda4c9ad42024cc23cf8191c64c5d8d55489cc199eb1022062 WatchSource:0}: Error finding container 721ff63903720cbda4c9ad42024cc23cf8191c64c5d8d55489cc199eb1022062: Status 404 returned error can't find the container with id 721ff63903720cbda4c9ad42024cc23cf8191c64c5d8d55489cc199eb1022062
Nov 02 15:40:05.914116 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:05.914078    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz" event={"ID":"a492f8b8-8322-402f-8803-fcd6735e897e","Type":"ContainerStarted","Data":"322475fe9bb3cb47f25dd52aecf39737e1a64b8fdb369cb51aca0293015e6647"}
Nov 02 15:40:05.914346 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:05.914135    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz" event={"ID":"a492f8b8-8322-402f-8803-fcd6735e897e","Type":"ContainerStarted","Data":"721ff63903720cbda4c9ad42024cc23cf8191c64c5d8d55489cc199eb1022062"}
Nov 02 15:40:06.036352 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:06.036269    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz" podStartSLOduration=2.036250382 podStartE2EDuration="2.036250382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:06.032752928 +0000 UTC m=+2613.625369727" watchObservedRunningTime="2026-11-02 15:40:06.036250382 +0000 UTC m=+2613.628867184"
Nov 02 15:40:06.922146 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:06.922109    4891 generic.go:356] "Generic (PLEG): container finished" podID="a492f8b8-8322-402f-8803-fcd6735e897e" containerID="322475fe9bb3cb47f25dd52aecf39737e1a64b8fdb369cb51aca0293015e6647" exitCode=0
Nov 02 15:40:06.922358 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:06.922189    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz" event={"ID":"a492f8b8-8322-402f-8803-fcd6735e897e","Type":"ContainerDied","Data":"322475fe9bb3cb47f25dd52aecf39737e1a64b8fdb369cb51aca0293015e6647"}
Nov 02 15:40:08.067052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.067026    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:08.162390 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.162360    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-openshift-ca\") pod \"a492f8b8-8322-402f-8803-fcd6735e897e\" (UID: \"a492f8b8-8322-402f-8803-fcd6735e897e\") "
Nov 02 15:40:08.162555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.162404    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-xks-ca\") pod \"a492f8b8-8322-402f-8803-fcd6735e897e\" (UID: \"a492f8b8-8322-402f-8803-fcd6735e897e\") "
Nov 02 15:40:08.162555 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.162471    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a492f8b8-8322-402f-8803-fcd6735e897e-kube-api-access-mlcn8\" (UniqueName: \"kubernetes.io/projected/a492f8b8-8322-402f-8803-fcd6735e897e-kube-api-access-mlcn8\") pod \"a492f8b8-8322-402f-8803-fcd6735e897e\" (UID: \"a492f8b8-8322-402f-8803-fcd6735e897e\") "
Nov 02 15:40:08.162775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.162748    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-openshift-ca" pod "a492f8b8-8322-402f-8803-fcd6735e897e" (UID: "a492f8b8-8322-402f-8803-fcd6735e897e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:40:08.162909 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.162893    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:08.164659 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.164633    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-xks-ca" pod "a492f8b8-8322-402f-8803-fcd6735e897e" (UID: "a492f8b8-8322-402f-8803-fcd6735e897e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:40:08.164761 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.164745    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a492f8b8-8322-402f-8803-fcd6735e897e-kube-api-access-mlcn8" pod "a492f8b8-8322-402f-8803-fcd6735e897e" (UID: "a492f8b8-8322-402f-8803-fcd6735e897e"). InnerVolumeSpecName "kube-api-access-mlcn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:08.180928 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.180872    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"]
Nov 02 15:40:08.183597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.183575    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68c393a9-rr58m"]
Nov 02 15:40:08.263812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.263776    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a492f8b8-8322-402f-8803-fcd6735e897e-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:08.263812 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.263807    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlcn8\" (UniqueName: \"kubernetes.io/projected/a492f8b8-8322-402f-8803-fcd6735e897e-kube-api-access-mlcn8\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:08.521656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.521628    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c068d1-4605-4d05-a8e0-a2661cd3831a" path="/var/lib/kubelet/pods/57c068d1-4605-4d05-a8e0-a2661cd3831a/volumes"
Nov 02 15:40:08.936440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.936354    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz" event={"ID":"a492f8b8-8322-402f-8803-fcd6735e897e","Type":"ContainerDied","Data":"721ff63903720cbda4c9ad42024cc23cf8191c64c5d8d55489cc199eb1022062"}
Nov 02 15:40:08.936440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.936378    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"
Nov 02 15:40:08.936440 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:08.936417    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721ff63903720cbda4c9ad42024cc23cf8191c64c5d8d55489cc199eb1022062"
Nov 02 15:40:09.185718 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.185692    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:40:09.272974 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.272947    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-envoy\") pod \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") "
Nov 02 15:40:09.273151 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273038    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-podinfo\") pod \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") "
Nov 02 15:40:09.273151 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273081    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istiod-ca-cert\") pod \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") "
Nov 02 15:40:09.273151 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273123    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-kube-api-access-wqjkh\" (UniqueName: \"kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-kube-api-access-wqjkh\") pod \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") "
Nov 02 15:40:09.273371 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273176    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-certs\") pod \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") "
Nov 02 15:40:09.273371 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273245    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-socket\") pod \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") "
Nov 02 15:40:09.273371 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273275    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-data\") pod \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") "
Nov 02 15:40:09.273371 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273361    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-credential-socket\") pod \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") "
Nov 02 15:40:09.273580 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273433    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-token\" (UniqueName: \"kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-token\") pod \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\" (UID: \"2a444b7c-3af1-40a3-aa72-2932b17a8a54\") "
Nov 02 15:40:09.273580 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273517    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-certs" pod "2a444b7c-3af1-40a3-aa72-2932b17a8a54" (UID: "2a444b7c-3af1-40a3-aa72-2932b17a8a54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:09.273691 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273588    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istiod-ca-cert" pod "2a444b7c-3af1-40a3-aa72-2932b17a8a54" (UID: "2a444b7c-3af1-40a3-aa72-2932b17a8a54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:40:09.273794 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273744    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-socket" pod "2a444b7c-3af1-40a3-aa72-2932b17a8a54" (UID: "2a444b7c-3af1-40a3-aa72-2932b17a8a54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:09.273869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273806    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-credential-socket" pod "2a444b7c-3af1-40a3-aa72-2932b17a8a54" (UID: "2a444b7c-3af1-40a3-aa72-2932b17a8a54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:09.273869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273821    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-data" pod "2a444b7c-3af1-40a3-aa72-2932b17a8a54" (UID: "2a444b7c-3af1-40a3-aa72-2932b17a8a54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:09.273963 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273946    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:09.274008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273969    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:09.274008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273982    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:09.274008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.273997    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:09.274143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.274009    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:09.275322 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.275288    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-kube-api-access-wqjkh" pod "2a444b7c-3af1-40a3-aa72-2932b17a8a54" (UID: "2a444b7c-3af1-40a3-aa72-2932b17a8a54"). InnerVolumeSpecName "kube-api-access-wqjkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:09.275408 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.275375    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-podinfo" pod "2a444b7c-3af1-40a3-aa72-2932b17a8a54" (UID: "2a444b7c-3af1-40a3-aa72-2932b17a8a54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:40:09.275971 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.275953    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-token" pod "2a444b7c-3af1-40a3-aa72-2932b17a8a54" (UID: "2a444b7c-3af1-40a3-aa72-2932b17a8a54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:09.276022 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.275956    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-envoy" pod "2a444b7c-3af1-40a3-aa72-2932b17a8a54" (UID: "2a444b7c-3af1-40a3-aa72-2932b17a8a54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:09.374707 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.374671    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:09.374707 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.374704    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqjkh\" (UniqueName: \"kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-kube-api-access-wqjkh\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:09.374920 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.374717    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:09.374920 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.374731    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a444b7c-3af1-40a3-aa72-2932b17a8a54-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:09.944954 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.944920    4891 generic.go:356] "Generic (PLEG): container finished" podID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerID="7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e" exitCode=0
Nov 02 15:40:09.945170 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.944972    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"
Nov 02 15:40:09.945170 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.945007    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" event={"ID":"2a444b7c-3af1-40a3-aa72-2932b17a8a54","Type":"ContainerDied","Data":"7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e"}
Nov 02 15:40:09.945170 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.945072    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" event={"ID":"2a444b7c-3af1-40a3-aa72-2932b17a8a54","Type":"ContainerDied","Data":"bf6b42118a5ac696de10c26584c1d10d0626e5369c8d84c0d3b2c28bc74698e5"}
Nov 02 15:40:09.945170 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.945105    4891 scope.go:122] "RemoveContainer" containerID="7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e"
Nov 02 15:40:09.959137 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.959123    4891 scope.go:122] "RemoveContainer" containerID="195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c"
Nov 02 15:40:09.968338 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.968294    4891 scope.go:122] "RemoveContainer" containerID="7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e"
Nov 02 15:40:09.968597 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:40:09.968577    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e\": container with ID starting with 7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e not found: ID does not exist" containerID="7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e"
Nov 02 15:40:09.968656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.968610    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e"} err="failed to get container status \"7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e\": rpc error: code = NotFound desc = could not find container \"7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e\": container with ID starting with 7c442c35c7730465bcdb050fd6a2d8ef564dba65d8229416586082ac83c9952e not found: ID does not exist"
Nov 02 15:40:09.968656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.968629    4891 scope.go:122] "RemoveContainer" containerID="195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c"
Nov 02 15:40:09.968898 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:40:09.968881    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c\": container with ID starting with 195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c not found: ID does not exist" containerID="195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c"
Nov 02 15:40:09.968944 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:09.968908    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c"} err="failed to get container status \"195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c\": rpc error: code = NotFound desc = could not find container \"195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c\": container with ID starting with 195a448ee683dfedb730b350bf47c19951f08f10e2dd62326294a7bb828d3c4c not found: ID does not exist"
Nov 02 15:40:10.111339 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:10.111293    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"]
Nov 02 15:40:10.114784 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:10.114762    4891 patch_prober.go:29] interesting pod/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:40:10.114860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:10.114819    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:40:10.160535 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:10.160497    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-39098c-openshift-default-76bff56b5f-b9c7h"]
Nov 02 15:40:10.521143 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:10.521108    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" path="/var/lib/kubelet/pods/2a444b7c-3af1-40a3-aa72-2932b17a8a54/volumes"
Nov 02 15:40:11.956780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:11.956721    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"
Nov 02 15:40:11.959502 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:11.959478    4891 generic.go:356] "Generic (PLEG): container finished" podID="cd686938-df40-4efe-ad36-53c6337f78e4" containerID="4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff" exitCode=0
Nov 02 15:40:11.959625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:11.959516    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"
Nov 02 15:40:11.959625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:11.959519    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x" event={"ID":"cd686938-df40-4efe-ad36-53c6337f78e4","Type":"ContainerDied","Data":"4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff"}
Nov 02 15:40:11.959625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:11.959571    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x" event={"ID":"cd686938-df40-4efe-ad36-53c6337f78e4","Type":"ContainerDied","Data":"9bf83fb57d9cc8550840c92157ff2742587a75685993caa5c585395c82580468"}
Nov 02 15:40:11.959625 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:11.959593    4891 scope.go:122] "RemoveContainer" containerID="4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff"
Nov 02 15:40:11.968969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:11.968949    4891 scope.go:122] "RemoveContainer" containerID="4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff"
Nov 02 15:40:11.969224 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:40:11.969207    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff\": container with ID starting with 4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff not found: ID does not exist" containerID="4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff"
Nov 02 15:40:11.969278 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:11.969233    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff"} err="failed to get container status \"4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff\": rpc error: code = NotFound desc = could not find container \"4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff\": container with ID starting with 4a3b6a110f991d75cc983092665525c16d345437cbfcb9f86b787377570b93ff not found: ID does not exist"
Nov 02 15:40:12.103127 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:12.103092    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd686938-df40-4efe-ad36-53c6337f78e4-kube-api-access-q7292\" (UniqueName: \"kubernetes.io/projected/cd686938-df40-4efe-ad36-53c6337f78e4-kube-api-access-q7292\") pod \"cd686938-df40-4efe-ad36-53c6337f78e4\" (UID: \"cd686938-df40-4efe-ad36-53c6337f78e4\") "
Nov 02 15:40:12.103295 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:12.103171    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd686938-df40-4efe-ad36-53c6337f78e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd686938-df40-4efe-ad36-53c6337f78e4-plugins-config-volume\") pod \"cd686938-df40-4efe-ad36-53c6337f78e4\" (UID: \"cd686938-df40-4efe-ad36-53c6337f78e4\") "
Nov 02 15:40:12.103622 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:12.103583    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd686938-df40-4efe-ad36-53c6337f78e4-plugins-config-volume" pod "cd686938-df40-4efe-ad36-53c6337f78e4" (UID: "cd686938-df40-4efe-ad36-53c6337f78e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:40:12.105281 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:12.105257    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd686938-df40-4efe-ad36-53c6337f78e4-kube-api-access-q7292" pod "cd686938-df40-4efe-ad36-53c6337f78e4" (UID: "cd686938-df40-4efe-ad36-53c6337f78e4"). InnerVolumeSpecName "kube-api-access-q7292". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:12.204810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:12.204776    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7292\" (UniqueName: \"kubernetes.io/projected/cd686938-df40-4efe-ad36-53c6337f78e4-kube-api-access-q7292\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:12.204810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:12.204806    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd686938-df40-4efe-ad36-53c6337f78e4-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:12.381853 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:12.381814    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"]
Nov 02 15:40:12.387659 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:12.387627    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-39098c-85df849f9f-7vp5x"]
Nov 02 15:40:12.520817 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:12.520793    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd686938-df40-4efe-ad36-53c6337f78e4" path="/var/lib/kubelet/pods/cd686938-df40-4efe-ad36-53c6337f78e4/volumes"
Nov 02 15:40:14.272482 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:40:14.272444    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:40:16.407677 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:16.407628    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"]
Nov 02 15:40:21.408815 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:21.408756    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb" podUID="24ccf1c1-6881-4b37-bf6e-883e6e26ef80" containerName="payload-pre-processing" containerID="cri-o://516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414" gracePeriod=25
Nov 02 15:40:23.152547 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:23.152508    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb" podUID="24ccf1c1-6881-4b37-bf6e-883e6e26ef80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:40:27.579963 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:27.579928    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"]
Nov 02 15:40:27.629100 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:27.629065    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4e4683-7ftvz"]
Nov 02 15:40:28.520786 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:28.520757    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a492f8b8-8322-402f-8803-fcd6735e897e" path="/var/lib/kubelet/pods/a492f8b8-8322-402f-8803-fcd6735e897e/volumes"
Nov 02 15:40:33.153279 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:33.153233    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb" podUID="24ccf1c1-6881-4b37-bf6e-883e6e26ef80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:40:35.686525 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.686487    4891 scope.go:122] "RemoveContainer" containerID="9957cd1bcb70ba57a8e0fb71f6d269682d136184e869dfac7cc4a857e23e4971"
Nov 02 15:40:35.696085 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.696062    4891 scope.go:122] "RemoveContainer" containerID="c461f8ebe044cf57a22d77ef624e1d2f4995cf66fffe14dc13833a059bb11908"
Nov 02 15:40:35.893684 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.893650    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"]
Nov 02 15:40:35.894094 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.894080    4891 cpu_manager.go:414] "containerMap: removing container" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy"
Nov 02 15:40:35.894141 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.894095    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy"
Nov 02 15:40:35.894141 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.894111    4891 cpu_manager.go:414] "containerMap: removing container" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy"
Nov 02 15:40:35.894141 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.894117    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy"
Nov 02 15:40:35.894141 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.894124    4891 cpu_manager.go:414] "containerMap: removing container" podUID="cd686938-df40-4efe-ad36-53c6337f78e4" containerName="payload-pre-processing"
Nov 02 15:40:35.894141 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.894129    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd686938-df40-4efe-ad36-53c6337f78e4" containerName="payload-pre-processing"
Nov 02 15:40:35.894307 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.894155    4891 cpu_manager.go:414] "containerMap: removing container" podUID="a492f8b8-8322-402f-8803-fcd6735e897e" containerName="revoke-keys"
Nov 02 15:40:35.894307 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.894160    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a492f8b8-8322-402f-8803-fcd6735e897e" containerName="revoke-keys"
Nov 02 15:40:35.894307 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.894224    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy"
Nov 02 15:40:35.894307 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.894232    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a444b7c-3af1-40a3-aa72-2932b17a8a54" containerName="istio-proxy"
Nov 02 15:40:35.894307 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.894237    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd686938-df40-4efe-ad36-53c6337f78e4" containerName="payload-pre-processing"
Nov 02 15:40:35.894307 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.894249    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="a492f8b8-8322-402f-8803-fcd6735e897e" containerName="revoke-keys"
Nov 02 15:40:35.898843 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.898823    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:35.932166 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.932140    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h\" (UID: \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:35.932300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.932183    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2jdp\" (UniqueName: \"kubernetes.io/projected/63c456f3-2e2e-4643-8e52-180b76aa0b1f-kube-api-access-q2jdp\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h\" (UID: \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:35.932300 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.932212    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h\" (UID: \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:35.962969 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:35.962895    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"]
Nov 02 15:40:36.008656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:36.008614    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:40:36.008821 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:36.008814    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:40:36.008869 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:36.008819    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:40:36.008907 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:36.008893    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:40:36.033078 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:36.033047    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h\" (UID: \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:36.033244 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:36.033094    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2jdp\" (UniqueName: \"kubernetes.io/projected/63c456f3-2e2e-4643-8e52-180b76aa0b1f-kube-api-access-q2jdp\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h\" (UID: \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:36.033244 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:36.033112    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h\" (UID: \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:36.033800 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:36.033778    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h\" (UID: \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:36.035470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:36.035451    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h\" (UID: \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:36.093781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:36.093751    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2jdp\" (UniqueName: \"kubernetes.io/projected/63c456f3-2e2e-4643-8e52-180b76aa0b1f-kube-api-access-q2jdp\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h\" (UID: \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:36.212830 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:36.212797    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:36.603210 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:36.603180    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"]
Nov 02 15:40:36.604522 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:40:36.604497    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c456f3_2e2e_4643_8e52_180b76aa0b1f.slice/crio-3ead51e9e459d1f26784757dc5976b8e815f55be3e660e4436d88d868d087040 WatchSource:0}: Error finding container 3ead51e9e459d1f26784757dc5976b8e815f55be3e660e4436d88d868d087040: Status 404 returned error can't find the container with id 3ead51e9e459d1f26784757dc5976b8e815f55be3e660e4436d88d868d087040
Nov 02 15:40:37.119482 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:37.119391    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h" event={"ID":"63c456f3-2e2e-4643-8e52-180b76aa0b1f","Type":"ContainerStarted","Data":"b67b4205265ccc0e793a2e5f79cbc33be7c7812ee5a5b3374cf4fcb5c322dcd0"}
Nov 02 15:40:37.119482 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:37.119446    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h" event={"ID":"63c456f3-2e2e-4643-8e52-180b76aa0b1f","Type":"ContainerStarted","Data":"3ead51e9e459d1f26784757dc5976b8e815f55be3e660e4436d88d868d087040"}
Nov 02 15:40:37.235888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:37.235823    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h" podStartSLOduration=2.23580517 podStartE2EDuration="2.23580517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:37.235452178 +0000 UTC m=+2644.828068980" watchObservedRunningTime="2026-11-02 15:40:37.23580517 +0000 UTC m=+2644.828421972"
Nov 02 15:40:38.128018 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:38.127981    4891 generic.go:356] "Generic (PLEG): container finished" podID="63c456f3-2e2e-4643-8e52-180b76aa0b1f" containerID="b67b4205265ccc0e793a2e5f79cbc33be7c7812ee5a5b3374cf4fcb5c322dcd0" exitCode=0
Nov 02 15:40:38.128498 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:38.128061    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h" event={"ID":"63c456f3-2e2e-4643-8e52-180b76aa0b1f","Type":"ContainerDied","Data":"b67b4205265ccc0e793a2e5f79cbc33be7c7812ee5a5b3374cf4fcb5c322dcd0"}
Nov 02 15:40:39.263351 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:39.263326    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:39.366612 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:39.366581    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c456f3-2e2e-4643-8e52-180b76aa0b1f-kube-api-access-q2jdp\" (UniqueName: \"kubernetes.io/projected/63c456f3-2e2e-4643-8e52-180b76aa0b1f-kube-api-access-q2jdp\") pod \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\" (UID: \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\") "
Nov 02 15:40:39.366792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:39.366621    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-openshift-ca\") pod \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\" (UID: \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\") "
Nov 02 15:40:39.366792 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:39.366675    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-xks-ca\") pod \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\" (UID: \"63c456f3-2e2e-4643-8e52-180b76aa0b1f\") "
Nov 02 15:40:39.367096 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:39.367064    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-openshift-ca" pod "63c456f3-2e2e-4643-8e52-180b76aa0b1f" (UID: "63c456f3-2e2e-4643-8e52-180b76aa0b1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:40:39.368814 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:39.368789    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-xks-ca" pod "63c456f3-2e2e-4643-8e52-180b76aa0b1f" (UID: "63c456f3-2e2e-4643-8e52-180b76aa0b1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:40:39.368940 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:39.368924    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c456f3-2e2e-4643-8e52-180b76aa0b1f-kube-api-access-q2jdp" pod "63c456f3-2e2e-4643-8e52-180b76aa0b1f" (UID: "63c456f3-2e2e-4643-8e52-180b76aa0b1f"). InnerVolumeSpecName "kube-api-access-q2jdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:39.467685 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:39.467604    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2jdp\" (UniqueName: \"kubernetes.io/projected/63c456f3-2e2e-4643-8e52-180b76aa0b1f-kube-api-access-q2jdp\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:39.467685 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:39.467631    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:39.467685 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:39.467648    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c456f3-2e2e-4643-8e52-180b76aa0b1f-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:40.142122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:40.142094    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"
Nov 02 15:40:40.142122 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:40.142110    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h" event={"ID":"63c456f3-2e2e-4643-8e52-180b76aa0b1f","Type":"ContainerDied","Data":"3ead51e9e459d1f26784757dc5976b8e815f55be3e660e4436d88d868d087040"}
Nov 02 15:40:40.142380 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:40.142166    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ead51e9e459d1f26784757dc5976b8e815f55be3e660e4436d88d868d087040"
Nov 02 15:40:41.659837 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:41.659812    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"
Nov 02 15:40:41.789321 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:41.789275    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-plugins-config-volume\") pod \"24ccf1c1-6881-4b37-bf6e-883e6e26ef80\" (UID: \"24ccf1c1-6881-4b37-bf6e-883e6e26ef80\") "
Nov 02 15:40:41.789488 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:41.789379    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-kube-api-access-jbd54\" (UniqueName: \"kubernetes.io/projected/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-kube-api-access-jbd54\") pod \"24ccf1c1-6881-4b37-bf6e-883e6e26ef80\" (UID: \"24ccf1c1-6881-4b37-bf6e-883e6e26ef80\") "
Nov 02 15:40:41.789733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:41.789710    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-plugins-config-volume" pod "24ccf1c1-6881-4b37-bf6e-883e6e26ef80" (UID: "24ccf1c1-6881-4b37-bf6e-883e6e26ef80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:40:41.791496 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:41.791463    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-kube-api-access-jbd54" pod "24ccf1c1-6881-4b37-bf6e-883e6e26ef80" (UID: "24ccf1c1-6881-4b37-bf6e-883e6e26ef80"). InnerVolumeSpecName "kube-api-access-jbd54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:41.891183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:41.891147    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbd54\" (UniqueName: \"kubernetes.io/projected/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-kube-api-access-jbd54\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:41.891183 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:41.891178    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ccf1c1-6881-4b37-bf6e-883e6e26ef80-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:42.155288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:42.155199    4891 generic.go:356] "Generic (PLEG): container finished" podID="24ccf1c1-6881-4b37-bf6e-883e6e26ef80" containerID="516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414" exitCode=0
Nov 02 15:40:42.155288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:42.155254    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"
Nov 02 15:40:42.155501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:42.155291    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb" event={"ID":"24ccf1c1-6881-4b37-bf6e-883e6e26ef80","Type":"ContainerDied","Data":"516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414"}
Nov 02 15:40:42.155501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:42.155369    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb" event={"ID":"24ccf1c1-6881-4b37-bf6e-883e6e26ef80","Type":"ContainerDied","Data":"991262d72ea1b48d00b67732e55f8577c2a380c9c87fce42c2afeae200f58820"}
Nov 02 15:40:42.155501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:42.155391    4891 scope.go:122] "RemoveContainer" containerID="516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414"
Nov 02 15:40:42.165585 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:42.165562    4891 scope.go:122] "RemoveContainer" containerID="516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414"
Nov 02 15:40:42.165816 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:40:42.165797    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414\": container with ID starting with 516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414 not found: ID does not exist" containerID="516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414"
Nov 02 15:40:42.165861 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:42.165826    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414"} err="failed to get container status \"516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414\": rpc error: code = NotFound desc = could not find container \"516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414\": container with ID starting with 516470bb70ca26aeb421f5aa8e22f2746344af22bbc01f920a2a552fc2cc2414 not found: ID does not exist"
Nov 02 15:40:42.321734 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:42.321700    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"]
Nov 02 15:40:42.327129 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:42.327102    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4e4683-66d784d5c5-9b5lb"]
Nov 02 15:40:42.521485 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:42.521451    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ccf1c1-6881-4b37-bf6e-883e6e26ef80" path="/var/lib/kubelet/pods/24ccf1c1-6881-4b37-bf6e-883e6e26ef80/volumes"
Nov 02 15:40:44.272835 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:40:44.272801    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:40:47.401206 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:47.401171    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"]
Nov 02 15:40:52.013052 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:52.013012    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"]
Nov 02 15:40:52.062697 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:52.062661    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-895ac8-dfg7h"]
Nov 02 15:40:52.401769 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:52.401709    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2" podUID="d97704bf-338e-4982-85ab-770f8aafdb5c" containerName="payload-pre-processing" containerID="cri-o://58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85" gracePeriod=25
Nov 02 15:40:52.544448 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:52.544403    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c456f3-2e2e-4643-8e52-180b76aa0b1f" path="/var/lib/kubelet/pods/63c456f3-2e2e-4643-8e52-180b76aa0b1f/volumes"
Nov 02 15:40:52.676781 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:52.676754    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"
Nov 02 15:40:52.804477 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:52.804445    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d97704bf-338e-4982-85ab-770f8aafdb5c-kube-api-access-v6m6s\" (UniqueName: \"kubernetes.io/projected/d97704bf-338e-4982-85ab-770f8aafdb5c-kube-api-access-v6m6s\") pod \"d97704bf-338e-4982-85ab-770f8aafdb5c\" (UID: \"d97704bf-338e-4982-85ab-770f8aafdb5c\") "
Nov 02 15:40:52.804646 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:52.804506    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d97704bf-338e-4982-85ab-770f8aafdb5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97704bf-338e-4982-85ab-770f8aafdb5c-plugins-config-volume\") pod \"d97704bf-338e-4982-85ab-770f8aafdb5c\" (UID: \"d97704bf-338e-4982-85ab-770f8aafdb5c\") "
Nov 02 15:40:52.804882 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:52.804859    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97704bf-338e-4982-85ab-770f8aafdb5c-plugins-config-volume" pod "d97704bf-338e-4982-85ab-770f8aafdb5c" (UID: "d97704bf-338e-4982-85ab-770f8aafdb5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:40:52.806617 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:52.806597    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97704bf-338e-4982-85ab-770f8aafdb5c-kube-api-access-v6m6s" pod "d97704bf-338e-4982-85ab-770f8aafdb5c" (UID: "d97704bf-338e-4982-85ab-770f8aafdb5c"). InnerVolumeSpecName "kube-api-access-v6m6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:52.906160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:52.906079    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6m6s\" (UniqueName: \"kubernetes.io/projected/d97704bf-338e-4982-85ab-770f8aafdb5c-kube-api-access-v6m6s\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:52.906160 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:52.906114    4891 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97704bf-338e-4982-85ab-770f8aafdb5c-plugins-config-volume\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:40:53.223233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:53.223143    4891 generic.go:356] "Generic (PLEG): container finished" podID="d97704bf-338e-4982-85ab-770f8aafdb5c" containerID="58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85" exitCode=0
Nov 02 15:40:53.223233 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:53.223200    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"
Nov 02 15:40:53.223739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:53.223229    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2" event={"ID":"d97704bf-338e-4982-85ab-770f8aafdb5c","Type":"ContainerDied","Data":"58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85"}
Nov 02 15:40:53.223739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:53.223287    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2" event={"ID":"d97704bf-338e-4982-85ab-770f8aafdb5c","Type":"ContainerDied","Data":"5cdfa6fbd00f4d7a41fa04d9a8e84cebd0d49f4aa354ace369bad9911dd93cfb"}
Nov 02 15:40:53.223739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:53.223326    4891 scope.go:122] "RemoveContainer" containerID="58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85"
Nov 02 15:40:53.233155 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:53.233132    4891 scope.go:122] "RemoveContainer" containerID="58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85"
Nov 02 15:40:53.233484 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:40:53.233460    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85\": container with ID starting with 58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85 not found: ID does not exist" containerID="58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85"
Nov 02 15:40:53.233564 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:53.233495    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85"} err="failed to get container status \"58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85\": rpc error: code = NotFound desc = could not find container \"58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85\": container with ID starting with 58075e4795eea41c95cb8addced6f9c797d601aa83747cbe313695895950ad85 not found: ID does not exist"
Nov 02 15:40:53.388003 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:53.387964    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"]
Nov 02 15:40:53.437558 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:53.437513    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-895ac8-65fcccbb9f-m5hp2"]
Nov 02 15:40:54.521129 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:40:54.521098    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d97704bf-338e-4982-85ab-770f8aafdb5c" path="/var/lib/kubelet/pods/d97704bf-338e-4982-85ab-770f8aafdb5c/volumes"
Nov 02 15:41:10.436185 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.436150    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"]
Nov 02 15:41:10.436650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.436563    4891 cpu_manager.go:414] "containerMap: removing container" podUID="24ccf1c1-6881-4b37-bf6e-883e6e26ef80" containerName="payload-pre-processing"
Nov 02 15:41:10.436650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.436575    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ccf1c1-6881-4b37-bf6e-883e6e26ef80" containerName="payload-pre-processing"
Nov 02 15:41:10.436650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.436589    4891 cpu_manager.go:414] "containerMap: removing container" podUID="63c456f3-2e2e-4643-8e52-180b76aa0b1f" containerName="revoke-keys"
Nov 02 15:41:10.436650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.436594    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c456f3-2e2e-4643-8e52-180b76aa0b1f" containerName="revoke-keys"
Nov 02 15:41:10.436650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.436621    4891 cpu_manager.go:414] "containerMap: removing container" podUID="d97704bf-338e-4982-85ab-770f8aafdb5c" containerName="payload-pre-processing"
Nov 02 15:41:10.436650 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.436627    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d97704bf-338e-4982-85ab-770f8aafdb5c" containerName="payload-pre-processing"
Nov 02 15:41:10.436860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.436683    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="d97704bf-338e-4982-85ab-770f8aafdb5c" containerName="payload-pre-processing"
Nov 02 15:41:10.436860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.436693    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ccf1c1-6881-4b37-bf6e-883e6e26ef80" containerName="payload-pre-processing"
Nov 02 15:41:10.436860 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.436700    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c456f3-2e2e-4643-8e52-180b76aa0b1f" containerName="revoke-keys"
Nov 02 15:41:10.439903 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.439887    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:10.479272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.479236    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv\" (UID: \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:10.479272 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.479276    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58w24\" (UniqueName: \"kubernetes.io/projected/3f5a3cbe-6261-4234-b1f3-460a662bb59d-kube-api-access-58w24\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv\" (UID: \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:10.479522 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.479302    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv\" (UID: \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:10.497886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.497853    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"]
Nov 02 15:41:10.544424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.544391    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:41:10.544424 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.544403    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:41:10.544655 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.544430    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:41:10.544655 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.544456    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:41:10.580456 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.580425    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv\" (UID: \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:10.580656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.580467    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58w24\" (UniqueName: \"kubernetes.io/projected/3f5a3cbe-6261-4234-b1f3-460a662bb59d-kube-api-access-58w24\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv\" (UID: \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:10.580656 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.580595    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv\" (UID: \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:10.581265 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.581247    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv\" (UID: \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:10.582894 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.582876    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv\" (UID: \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:10.641810 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.641772    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58w24\" (UniqueName: \"kubernetes.io/projected/3f5a3cbe-6261-4234-b1f3-460a662bb59d-kube-api-access-58w24\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv\" (UID: \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:10.753805 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:10.753726    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:11.130946 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:11.130917    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"]
Nov 02 15:41:11.133195 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:41:11.133166    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5a3cbe_6261_4234_b1f3_460a662bb59d.slice/crio-217e4a18c73bdf1b80c91638774858dec34d16c1c0af1a6450a2c85e5d929a04 WatchSource:0}: Error finding container 217e4a18c73bdf1b80c91638774858dec34d16c1c0af1a6450a2c85e5d929a04: Status 404 returned error can't find the container with id 217e4a18c73bdf1b80c91638774858dec34d16c1c0af1a6450a2c85e5d929a04
Nov 02 15:41:11.351159 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:11.351127    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv" event={"ID":"3f5a3cbe-6261-4234-b1f3-460a662bb59d","Type":"ContainerStarted","Data":"217e4a18c73bdf1b80c91638774858dec34d16c1c0af1a6450a2c85e5d929a04"}
Nov 02 15:41:12.358517 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:12.358433    4891 generic.go:356] "Generic (PLEG): container finished" podID="3f5a3cbe-6261-4234-b1f3-460a662bb59d" containerID="402efe581ba2894ca98663d66d70c08384e173fd75436f6b4609277f8497ae4a" exitCode=0
Nov 02 15:41:12.358884 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:12.358518    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv" event={"ID":"3f5a3cbe-6261-4234-b1f3-460a662bb59d","Type":"ContainerDied","Data":"402efe581ba2894ca98663d66d70c08384e173fd75436f6b4609277f8497ae4a"}
Nov 02 15:41:13.499643 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:13.499612    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:13.609494 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:13.609463    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-openshift-ca\") pod \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\" (UID: \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\") "
Nov 02 15:41:13.609671 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:13.609597    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-xks-ca\") pod \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\" (UID: \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\") "
Nov 02 15:41:13.609671 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:13.609648    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5a3cbe-6261-4234-b1f3-460a662bb59d-kube-api-access-58w24\" (UniqueName: \"kubernetes.io/projected/3f5a3cbe-6261-4234-b1f3-460a662bb59d-kube-api-access-58w24\") pod \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\" (UID: \"3f5a3cbe-6261-4234-b1f3-460a662bb59d\") "
Nov 02 15:41:13.609888 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:13.609862    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-openshift-ca" pod "3f5a3cbe-6261-4234-b1f3-460a662bb59d" (UID: "3f5a3cbe-6261-4234-b1f3-460a662bb59d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:41:13.609992 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:13.609970    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:13.611745 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:13.611715    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-xks-ca" pod "3f5a3cbe-6261-4234-b1f3-460a662bb59d" (UID: "3f5a3cbe-6261-4234-b1f3-460a662bb59d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:41:13.611864 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:13.611787    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5a3cbe-6261-4234-b1f3-460a662bb59d-kube-api-access-58w24" pod "3f5a3cbe-6261-4234-b1f3-460a662bb59d" (UID: "3f5a3cbe-6261-4234-b1f3-460a662bb59d"). InnerVolumeSpecName "kube-api-access-58w24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:41:13.710905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:13.710867    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5a3cbe-6261-4234-b1f3-460a662bb59d-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:13.710905 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:13.710897    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58w24\" (UniqueName: \"kubernetes.io/projected/3f5a3cbe-6261-4234-b1f3-460a662bb59d-kube-api-access-58w24\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:14.273346 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:41:14.273282    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:41:14.371053 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:14.371021    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"
Nov 02 15:41:14.371053 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:14.371043    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv" event={"ID":"3f5a3cbe-6261-4234-b1f3-460a662bb59d","Type":"ContainerDied","Data":"217e4a18c73bdf1b80c91638774858dec34d16c1c0af1a6450a2c85e5d929a04"}
Nov 02 15:41:14.371273 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:14.371092    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="217e4a18c73bdf1b80c91638774858dec34d16c1c0af1a6450a2c85e5d929a04"
Nov 02 15:41:27.106803 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:27.106757    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"]
Nov 02 15:41:27.154910 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:27.154872    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fd1339-jq9mv"]
Nov 02 15:41:28.520488 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:28.520457    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5a3cbe-6261-4234-b1f3-460a662bb59d" path="/var/lib/kubelet/pods/3f5a3cbe-6261-4234-b1f3-460a662bb59d/volumes"
Nov 02 15:41:33.283159 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:33.283128    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:41:33.284184 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:33.284160    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:41:33.286758 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:33.286736    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:41:33.287685 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:33.287665    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:41:33.298428 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:33.298414    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:41:33.302282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:33.302263    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:41:35.692590 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.692555    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"]
Nov 02 15:41:35.692998 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.692983    4891 cpu_manager.go:414] "containerMap: removing container" podUID="3f5a3cbe-6261-4234-b1f3-460a662bb59d" containerName="revoke-keys"
Nov 02 15:41:35.693043 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.692999    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5a3cbe-6261-4234-b1f3-460a662bb59d" containerName="revoke-keys"
Nov 02 15:41:35.693080 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.693060    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5a3cbe-6261-4234-b1f3-460a662bb59d" containerName="revoke-keys"
Nov 02 15:41:35.696356 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.696338    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:35.743796 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.743766    4891 scope.go:122] "RemoveContainer" containerID="24f3bb8617fcdda1c0bfb0a80d850b2d49fe0956378950252c49f6a9d24d2808"
Nov 02 15:41:35.748723 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.748695    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:41:35.757764 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.757737    4891 scope.go:122] "RemoveContainer" containerID="bbe0f7131c383d63441022f05c9a14b0d3971c4a78c108b7dfca5293a1f06d04"
Nov 02 15:41:35.769069 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.769044    4891 scope.go:122] "RemoveContainer" containerID="58c6a60485788006268d185afa8ebd983491e0ca142f65931d81bb3f1dbabb3c"
Nov 02 15:41:35.798395 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.798373    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:41:35.798513 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.798433    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:41:35.798513 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.798476    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:41:35.801426 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.801407    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"]
Nov 02 15:41:35.828139 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.828108    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2txh\" (UniqueName: \"kubernetes.io/projected/0eff832d-e4cf-48c8-a115-bbc2b597327a-kube-api-access-h2txh\") pod \"maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk\" (UID: \"0eff832d-e4cf-48c8-a115-bbc2b597327a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:35.828262 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.828159    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk\" (UID: \"0eff832d-e4cf-48c8-a115-bbc2b597327a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:35.828345 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.828264    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk\" (UID: \"0eff832d-e4cf-48c8-a115-bbc2b597327a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:35.928950 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.928911    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2txh\" (UniqueName: \"kubernetes.io/projected/0eff832d-e4cf-48c8-a115-bbc2b597327a-kube-api-access-h2txh\") pod \"maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk\" (UID: \"0eff832d-e4cf-48c8-a115-bbc2b597327a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:35.928950 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.928956    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk\" (UID: \"0eff832d-e4cf-48c8-a115-bbc2b597327a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:35.929187 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.928984    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk\" (UID: \"0eff832d-e4cf-48c8-a115-bbc2b597327a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:35.929593 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.929571    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk\" (UID: \"0eff832d-e4cf-48c8-a115-bbc2b597327a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:35.931277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.931258    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk\" (UID: \"0eff832d-e4cf-48c8-a115-bbc2b597327a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:35.992703 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:35.992628    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2txh\" (UniqueName: \"kubernetes.io/projected/0eff832d-e4cf-48c8-a115-bbc2b597327a-kube-api-access-h2txh\") pod \"maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk\" (UID: \"0eff832d-e4cf-48c8-a115-bbc2b597327a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:36.009470 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:36.009443    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:36.388476 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:36.388447    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"]
Nov 02 15:41:36.390103 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:41:36.390071    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eff832d_e4cf_48c8_a115_bbc2b597327a.slice/crio-033a07943c76abef7f19c2f03055ddf048f4618f78810d88a0aa06d0371cb8ab WatchSource:0}: Error finding container 033a07943c76abef7f19c2f03055ddf048f4618f78810d88a0aa06d0371cb8ab: Status 404 returned error can't find the container with id 033a07943c76abef7f19c2f03055ddf048f4618f78810d88a0aa06d0371cb8ab
Nov 02 15:41:36.507048 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:36.506997    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk" event={"ID":"0eff832d-e4cf-48c8-a115-bbc2b597327a","Type":"ContainerStarted","Data":"033a07943c76abef7f19c2f03055ddf048f4618f78810d88a0aa06d0371cb8ab"}
Nov 02 15:41:37.514951 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:37.514867    4891 generic.go:356] "Generic (PLEG): container finished" podID="0eff832d-e4cf-48c8-a115-bbc2b597327a" containerID="8c4af0f1b404e86249850755662b949ebba6e6e885c55ea36cb6e2f11b0a656c" exitCode=0
Nov 02 15:41:37.515352 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:37.514956    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk" event={"ID":"0eff832d-e4cf-48c8-a115-bbc2b597327a","Type":"ContainerDied","Data":"8c4af0f1b404e86249850755662b949ebba6e6e885c55ea36cb6e2f11b0a656c"}
Nov 02 15:41:38.655269 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:38.655243    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:38.757431 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:38.757395    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eff832d-e4cf-48c8-a115-bbc2b597327a-kube-api-access-h2txh\" (UniqueName: \"kubernetes.io/projected/0eff832d-e4cf-48c8-a115-bbc2b597327a-kube-api-access-h2txh\") pod \"0eff832d-e4cf-48c8-a115-bbc2b597327a\" (UID: \"0eff832d-e4cf-48c8-a115-bbc2b597327a\") "
Nov 02 15:41:38.757634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:38.757461    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-openshift-ca\") pod \"0eff832d-e4cf-48c8-a115-bbc2b597327a\" (UID: \"0eff832d-e4cf-48c8-a115-bbc2b597327a\") "
Nov 02 15:41:38.757634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:38.757612    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-xks-ca\") pod \"0eff832d-e4cf-48c8-a115-bbc2b597327a\" (UID: \"0eff832d-e4cf-48c8-a115-bbc2b597327a\") "
Nov 02 15:41:38.757868 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:38.757837    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-openshift-ca" pod "0eff832d-e4cf-48c8-a115-bbc2b597327a" (UID: "0eff832d-e4cf-48c8-a115-bbc2b597327a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:41:38.759547 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:38.759517    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-xks-ca" pod "0eff832d-e4cf-48c8-a115-bbc2b597327a" (UID: "0eff832d-e4cf-48c8-a115-bbc2b597327a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:41:38.759668 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:38.759626    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eff832d-e4cf-48c8-a115-bbc2b597327a-kube-api-access-h2txh" pod "0eff832d-e4cf-48c8-a115-bbc2b597327a" (UID: "0eff832d-e4cf-48c8-a115-bbc2b597327a"). InnerVolumeSpecName "kube-api-access-h2txh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:41:38.858554 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:38.858477    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:38.858554 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:38.858506    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2txh\" (UniqueName: \"kubernetes.io/projected/0eff832d-e4cf-48c8-a115-bbc2b597327a-kube-api-access-h2txh\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:38.858554 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:38.858520    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eff832d-e4cf-48c8-a115-bbc2b597327a-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:39.528683 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:39.528645    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"
Nov 02 15:41:39.528881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:39.528691    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk" event={"ID":"0eff832d-e4cf-48c8-a115-bbc2b597327a","Type":"ContainerDied","Data":"033a07943c76abef7f19c2f03055ddf048f4618f78810d88a0aa06d0371cb8ab"}
Nov 02 15:41:39.528881 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:39.528743    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="033a07943c76abef7f19c2f03055ddf048f4618f78810d88a0aa06d0371cb8ab"
Nov 02 15:41:40.104597 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:40.104562    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"]
Nov 02 15:41:40.130916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:40.130882    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"]
Nov 02 15:41:43.049796 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:43.049754    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"]
Nov 02 15:41:43.099288 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:43.099252    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1575e9c-44xkl"]
Nov 02 15:41:44.273463 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:41:44.273427    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:41:44.520698 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:44.520664    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e" path="/var/lib/kubelet/pods/bfa7e1c5-5e70-4e7d-b8b7-6c81af30014e/volumes"
Nov 02 15:41:49.364090 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.364053    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"]
Nov 02 15:41:49.364520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.364466    4891 cpu_manager.go:414] "containerMap: removing container" podUID="0eff832d-e4cf-48c8-a115-bbc2b597327a" containerName="revoke-keys"
Nov 02 15:41:49.364520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.364480    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eff832d-e4cf-48c8-a115-bbc2b597327a" containerName="revoke-keys"
Nov 02 15:41:49.364596 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.364558    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eff832d-e4cf-48c8-a115-bbc2b597327a" containerName="revoke-keys"
Nov 02 15:41:49.368842 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.368825    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:49.423245 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.423217    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:41:49.453579 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.453548    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b447c72a-ctpkm\" (UID: \"cf48e4cf-9553-45d6-8213-3bc9b614b631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:49.453787 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.453627    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b447c72a-ctpkm\" (UID: \"cf48e4cf-9553-45d6-8213-3bc9b614b631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:49.453787 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.453668    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxjqm\" (UniqueName: \"kubernetes.io/projected/cf48e4cf-9553-45d6-8213-3bc9b614b631-kube-api-access-rxjqm\") pod \"maas-api-revoke-sub-b447c72a-ctpkm\" (UID: \"cf48e4cf-9553-45d6-8213-3bc9b614b631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:49.472804 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.472767    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:41:49.472974 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.472810    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:41:49.472974 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.472895    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 15:41:49.475519 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.475498    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"]
Nov 02 15:41:49.554248 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.554207    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b447c72a-ctpkm\" (UID: \"cf48e4cf-9553-45d6-8213-3bc9b614b631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:49.554463 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.554275    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxjqm\" (UniqueName: \"kubernetes.io/projected/cf48e4cf-9553-45d6-8213-3bc9b614b631-kube-api-access-rxjqm\") pod \"maas-api-revoke-sub-b447c72a-ctpkm\" (UID: \"cf48e4cf-9553-45d6-8213-3bc9b614b631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:49.554463 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.554386    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b447c72a-ctpkm\" (UID: \"cf48e4cf-9553-45d6-8213-3bc9b614b631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:49.555022 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.555000    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b447c72a-ctpkm\" (UID: \"cf48e4cf-9553-45d6-8213-3bc9b614b631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:49.556721 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.556698    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b447c72a-ctpkm\" (UID: \"cf48e4cf-9553-45d6-8213-3bc9b614b631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:49.615659 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.615585    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxjqm\" (UniqueName: \"kubernetes.io/projected/cf48e4cf-9553-45d6-8213-3bc9b614b631-kube-api-access-rxjqm\") pod \"maas-api-revoke-sub-b447c72a-ctpkm\" (UID: \"cf48e4cf-9553-45d6-8213-3bc9b614b631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:49.681725 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:49.681691    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:50.062374 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:50.062337    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"]
Nov 02 15:41:50.063822 ip-10-0-132-27 kubenswrapper[4891]: W1102 15:41:50.063798    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf48e4cf_9553_45d6_8213_3bc9b614b631.slice/crio-1c8d4c138388e6aa59106efaeed397107d077a1aef3bb317a8c86be4700d5427 WatchSource:0}: Error finding container 1c8d4c138388e6aa59106efaeed397107d077a1aef3bb317a8c86be4700d5427: Status 404 returned error can't find the container with id 1c8d4c138388e6aa59106efaeed397107d077a1aef3bb317a8c86be4700d5427
Nov 02 15:41:50.595017 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:50.594979    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm" event={"ID":"cf48e4cf-9553-45d6-8213-3bc9b614b631","Type":"ContainerStarted","Data":"f09979907bab6cbc567b1822706c720a9c54c247cbc0dba50771b4efc993de80"}
Nov 02 15:41:50.595521 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:50.595032    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm" event={"ID":"cf48e4cf-9553-45d6-8213-3bc9b614b631","Type":"ContainerStarted","Data":"1c8d4c138388e6aa59106efaeed397107d077a1aef3bb317a8c86be4700d5427"}
Nov 02 15:41:50.712620 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:50.712514    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm" podStartSLOduration=1.71249376 podStartE2EDuration="1.71249376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:41:50.711011081 +0000 UTC m=+2718.303627879" watchObservedRunningTime="2026-11-02 15:41:50.71249376 +0000 UTC m=+2718.305110560"
Nov 02 15:41:51.602542 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:51.602501    4891 generic.go:356] "Generic (PLEG): container finished" podID="cf48e4cf-9553-45d6-8213-3bc9b614b631" containerID="f09979907bab6cbc567b1822706c720a9c54c247cbc0dba50771b4efc993de80" exitCode=0
Nov 02 15:41:51.603024 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:51.602584    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm" event={"ID":"cf48e4cf-9553-45d6-8213-3bc9b614b631","Type":"ContainerDied","Data":"f09979907bab6cbc567b1822706c720a9c54c247cbc0dba50771b4efc993de80"}
Nov 02 15:41:52.746774 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:52.746744    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:52.786002 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:52.785973    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf48e4cf-9553-45d6-8213-3bc9b614b631-kube-api-access-rxjqm\" (UniqueName: \"kubernetes.io/projected/cf48e4cf-9553-45d6-8213-3bc9b614b631-kube-api-access-rxjqm\") pod \"cf48e4cf-9553-45d6-8213-3bc9b614b631\" (UID: \"cf48e4cf-9553-45d6-8213-3bc9b614b631\") "
Nov 02 15:41:52.786167 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:52.786026    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-openshift-ca\") pod \"cf48e4cf-9553-45d6-8213-3bc9b614b631\" (UID: \"cf48e4cf-9553-45d6-8213-3bc9b614b631\") "
Nov 02 15:41:52.786167 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:52.786054    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-xks-ca\") pod \"cf48e4cf-9553-45d6-8213-3bc9b614b631\" (UID: \"cf48e4cf-9553-45d6-8213-3bc9b614b631\") "
Nov 02 15:41:52.786428 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:52.786403    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-openshift-ca" pod "cf48e4cf-9553-45d6-8213-3bc9b614b631" (UID: "cf48e4cf-9553-45d6-8213-3bc9b614b631"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:41:52.788105 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:52.788085    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-xks-ca" pod "cf48e4cf-9553-45d6-8213-3bc9b614b631" (UID: "cf48e4cf-9553-45d6-8213-3bc9b614b631"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:41:52.788247 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:52.788226    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf48e4cf-9553-45d6-8213-3bc9b614b631-kube-api-access-rxjqm" pod "cf48e4cf-9553-45d6-8213-3bc9b614b631" (UID: "cf48e4cf-9553-45d6-8213-3bc9b614b631"). InnerVolumeSpecName "kube-api-access-rxjqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:41:52.886833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:52.886754    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxjqm\" (UniqueName: \"kubernetes.io/projected/cf48e4cf-9553-45d6-8213-3bc9b614b631-kube-api-access-rxjqm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:52.886833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:52.886788    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:52.886833 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:52.886798    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf48e4cf-9553-45d6-8213-3bc9b614b631-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:53.616209 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:53.616180    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"
Nov 02 15:41:53.616447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:53.616172    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm" event={"ID":"cf48e4cf-9553-45d6-8213-3bc9b614b631","Type":"ContainerDied","Data":"1c8d4c138388e6aa59106efaeed397107d077a1aef3bb317a8c86be4700d5427"}
Nov 02 15:41:53.616447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:53.616306    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c8d4c138388e6aa59106efaeed397107d077a1aef3bb317a8c86be4700d5427"
Nov 02 15:41:55.124730 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.124684    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main" containerID="cri-o://a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0" gracePeriod=45
Nov 02 15:41:55.150055 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.149990    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main" containerID="cri-o://64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e" gracePeriod=45
Nov 02 15:41:55.442282 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.442239    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:41:55.445472 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.445447    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:41:55.511613 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.511577    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-home\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-home\") pod \"4da5799a-02da-4d12-98ae-7eb568707007\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") "
Nov 02 15:41:55.511851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.511665    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da5799a-02da-4d12-98ae-7eb568707007-kube-api-access-94mff\" (UniqueName: \"kubernetes.io/projected/4da5799a-02da-4d12-98ae-7eb568707007-kube-api-access-94mff\") pod \"4da5799a-02da-4d12-98ae-7eb568707007\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") "
Nov 02 15:41:55.511851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.511712    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4da5799a-02da-4d12-98ae-7eb568707007-tls-certs\" (UniqueName: \"kubernetes.io/secret/4da5799a-02da-4d12-98ae-7eb568707007-tls-certs\") pod \"4da5799a-02da-4d12-98ae-7eb568707007\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") "
Nov 02 15:41:55.511851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.511762    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-dshm\") pod \"4da5799a-02da-4d12-98ae-7eb568707007\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") "
Nov 02 15:41:55.511851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.511837    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-model-cache\") pod \"4da5799a-02da-4d12-98ae-7eb568707007\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") "
Nov 02 15:41:55.512124 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.511888    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tmp-dir\") pod \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") "
Nov 02 15:41:55.512124 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.511890    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-home" pod "4da5799a-02da-4d12-98ae-7eb568707007" (UID: "4da5799a-02da-4d12-98ae-7eb568707007"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:41:55.512124 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.511920    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-tmp-dir\") pod \"4da5799a-02da-4d12-98ae-7eb568707007\" (UID: \"4da5799a-02da-4d12-98ae-7eb568707007\") "
Nov 02 15:41:55.512124 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.511969    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-home\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-home\") pod \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") "
Nov 02 15:41:55.512124 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512025    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-model-cache\") pod \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") "
Nov 02 15:41:55.512124 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512065    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tls-certs\") pod \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") "
Nov 02 15:41:55.512124 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512116    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-kube-api-access-c7nd7\" (UniqueName: \"kubernetes.io/projected/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-kube-api-access-c7nd7\") pod \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") "
Nov 02 15:41:55.512549 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512130    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-model-cache" pod "4da5799a-02da-4d12-98ae-7eb568707007" (UID: "4da5799a-02da-4d12-98ae-7eb568707007"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:41:55.512549 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512144    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-dshm\") pod \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\" (UID: \"fd4f9a33-e005-4bd4-97a4-e65d83ef4220\") "
Nov 02 15:41:55.512549 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512417    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-home" pod "fd4f9a33-e005-4bd4-97a4-e65d83ef4220" (UID: "fd4f9a33-e005-4bd4-97a4-e65d83ef4220"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:41:55.512549 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512444    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tmp-dir" pod "fd4f9a33-e005-4bd4-97a4-e65d83ef4220" (UID: "fd4f9a33-e005-4bd4-97a4-e65d83ef4220"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:41:55.512771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512633    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:55.512771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512654    4891 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-home\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:55.512771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512668    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:55.512771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512683    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:55.512771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512614    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-tmp-dir" pod "4da5799a-02da-4d12-98ae-7eb568707007" (UID: "4da5799a-02da-4d12-98ae-7eb568707007"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:41:55.512771 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.512686    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-model-cache" pod "fd4f9a33-e005-4bd4-97a4-e65d83ef4220" (UID: "fd4f9a33-e005-4bd4-97a4-e65d83ef4220"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:41:55.515739 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.515715    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da5799a-02da-4d12-98ae-7eb568707007-kube-api-access-94mff" pod "4da5799a-02da-4d12-98ae-7eb568707007" (UID: "4da5799a-02da-4d12-98ae-7eb568707007"). InnerVolumeSpecName "kube-api-access-94mff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:41:55.515886 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.515860    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-dshm" pod "fd4f9a33-e005-4bd4-97a4-e65d83ef4220" (UID: "fd4f9a33-e005-4bd4-97a4-e65d83ef4220"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:41:55.516008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.515990    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-kube-api-access-c7nd7" pod "fd4f9a33-e005-4bd4-97a4-e65d83ef4220" (UID: "fd4f9a33-e005-4bd4-97a4-e65d83ef4220"). InnerVolumeSpecName "kube-api-access-c7nd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:41:55.516352 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.516336    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-dshm" pod "4da5799a-02da-4d12-98ae-7eb568707007" (UID: "4da5799a-02da-4d12-98ae-7eb568707007"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:41:55.516733 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.516709    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tls-certs" pod "fd4f9a33-e005-4bd4-97a4-e65d83ef4220" (UID: "fd4f9a33-e005-4bd4-97a4-e65d83ef4220"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:41:55.516809 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.516733    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4da5799a-02da-4d12-98ae-7eb568707007-tls-certs" pod "4da5799a-02da-4d12-98ae-7eb568707007" (UID: "4da5799a-02da-4d12-98ae-7eb568707007"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:41:55.613565 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.613521    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da5799a-02da-4d12-98ae-7eb568707007-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:55.613565 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.613567    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:55.613775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.613581    4891 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4da5799a-02da-4d12-98ae-7eb568707007-tmp-dir\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:55.613775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.613595    4891 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-model-cache\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:55.613775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.613612    4891 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-tls-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:55.613775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.613626    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7nd7\" (UniqueName: \"kubernetes.io/projected/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-kube-api-access-c7nd7\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:55.613775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.613638    4891 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd4f9a33-e005-4bd4-97a4-e65d83ef4220-dshm\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:55.613775 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.613652    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94mff\" (UniqueName: \"kubernetes.io/projected/4da5799a-02da-4d12-98ae-7eb568707007-kube-api-access-94mff\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:41:55.638806 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.638770    4891 generic.go:356] "Generic (PLEG): container finished" podID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerID="64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e" exitCode=0
Nov 02 15:41:55.639012 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.638848    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"
Nov 02 15:41:55.639012 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.638875    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" event={"ID":"fd4f9a33-e005-4bd4-97a4-e65d83ef4220","Type":"ContainerDied","Data":"64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e"}
Nov 02 15:41:55.639012 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.638926    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv" event={"ID":"fd4f9a33-e005-4bd4-97a4-e65d83ef4220","Type":"ContainerDied","Data":"7023af2f7b7a0f3f7171e563f41ec8dea6a814a43b986b391084f848d4c6cc02"}
Nov 02 15:41:55.639012 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.638994    4891 scope.go:122] "RemoveContainer" containerID="64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e"
Nov 02 15:41:55.640927 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.640901    4891 generic.go:356] "Generic (PLEG): container finished" podID="4da5799a-02da-4d12-98ae-7eb568707007" containerID="a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0" exitCode=0
Nov 02 15:41:55.641131 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.641062    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" event={"ID":"4da5799a-02da-4d12-98ae-7eb568707007","Type":"ContainerDied","Data":"a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0"}
Nov 02 15:41:55.641358 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.641299    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn" event={"ID":"4da5799a-02da-4d12-98ae-7eb568707007","Type":"ContainerDied","Data":"bb2eaf51ed39abb2048c19a31a7c222efa081361f95d07904d4f1bc7ac967009"}
Nov 02 15:41:55.641501 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.641481    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"
Nov 02 15:41:55.651453 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.651396    4891 scope.go:122] "RemoveContainer" containerID="64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e"
Nov 02 15:41:55.651809 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:41:55.651780    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e\": container with ID starting with 64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e not found: ID does not exist" containerID="64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e"
Nov 02 15:41:55.651916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.651821    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e"} err="failed to get container status \"64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e\": rpc error: code = NotFound desc = could not find container \"64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e\": container with ID starting with 64d57ecd0b829376df97c35a59df3c22d5bf61b845238ef4bb9b6daf4d34310e not found: ID does not exist"
Nov 02 15:41:55.651916 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.651845    4891 scope.go:122] "RemoveContainer" containerID="a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0"
Nov 02 15:41:55.689999 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.689946    4891 scope.go:122] "RemoveContainer" containerID="a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0"
Nov 02 15:41:55.690404 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:41:55.690361    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0\": container with ID starting with a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0 not found: ID does not exist" containerID="a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0"
Nov 02 15:41:55.690504 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.690411    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0"} err="failed to get container status \"a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0\": rpc error: code = NotFound desc = could not find container \"a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0\": container with ID starting with a0dd37e78e5b7d5a8e9a0d922163b109e684a27cb28db0830b9a60045d0c57c0 not found: ID does not exist"
Nov 02 15:41:55.829077 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.829035    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"]
Nov 02 15:41:55.837489 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.837455    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/simulator-w1-019efe-kserve-56f7ff8c88-r9kvv"]
Nov 02 15:41:55.945851 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.945758    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"]
Nov 02 15:41:55.950416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:55.950386    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-019efe/distinct-2-w1-019efe-kserve-6d85dd5d55-dkmkn"]
Nov 02 15:41:56.520944 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:56.520907    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4da5799a-02da-4d12-98ae-7eb568707007" path="/var/lib/kubelet/pods/4da5799a-02da-4d12-98ae-7eb568707007/volumes"
Nov 02 15:41:56.521580 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:41:56.521560    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" path="/var/lib/kubelet/pods/fd4f9a33-e005-4bd4-97a4-e65d83ef4220/volumes"
Nov 02 15:42:06.927844 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:06.927807    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"]
Nov 02 15:42:06.977079 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:06.977048    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-019efe-4fswk"]
Nov 02 15:42:08.520835 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:08.520797    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eff832d-e4cf-48c8-a115-bbc2b597327a" path="/var/lib/kubelet/pods/0eff832d-e4cf-48c8-a115-bbc2b597327a/volumes"
Nov 02 15:42:14.059976 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:14.059900    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"]
Nov 02 15:42:14.060416 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:14.060089    4891 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerName="istio-proxy" containerID="cri-o://7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c" gracePeriod=30
Nov 02 15:42:14.273292 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:42:14.273255    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:42:15.000169 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:15.000134    4891 patch_prober.go:29] interesting pod/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:42:15.000363 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:15.000201    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:42:17.054608 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:17.054574    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"]
Nov 02 15:42:17.103863 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:17.103830    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3bb5b8-lv249"]
Nov 02 15:42:18.520277 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:18.520245    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d75071d-68cc-47e2-ae69-2adfb61f61c6" path="/var/lib/kubelet/pods/9d75071d-68cc-47e2-ae69-2adfb61f61c6/volumes"
Nov 02 15:42:19.309676 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.309651    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:42:19.452957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.452870    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-credential-socket\") pod \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") "
Nov 02 15:42:19.452957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.452920    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-kube-api-access-ztrjj\" (UniqueName: \"kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-kube-api-access-ztrjj\") pod \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") "
Nov 02 15:42:19.452957 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.452946    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-token\" (UniqueName: \"kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-token\") pod \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") "
Nov 02 15:42:19.453263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.452985    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-socket\") pod \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") "
Nov 02 15:42:19.453263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453033    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-certs\") pod \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") "
Nov 02 15:42:19.453263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453075    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-podinfo\") pod \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") "
Nov 02 15:42:19.453263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453099    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istiod-ca-cert\") pod \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") "
Nov 02 15:42:19.453263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453169    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-envoy\") pod \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") "
Nov 02 15:42:19.453263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453194    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-credential-socket" pod "f1cdffd6-0920-40c6-b265-dcd53bdbc95f" (UID: "f1cdffd6-0920-40c6-b265-dcd53bdbc95f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:42:19.453263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453216    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-data\") pod \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\" (UID: \"f1cdffd6-0920-40c6-b265-dcd53bdbc95f\") "
Nov 02 15:42:19.453735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453373    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-certs" pod "f1cdffd6-0920-40c6-b265-dcd53bdbc95f" (UID: "f1cdffd6-0920-40c6-b265-dcd53bdbc95f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:42:19.453735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453382    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-socket" pod "f1cdffd6-0920-40c6-b265-dcd53bdbc95f" (UID: "f1cdffd6-0920-40c6-b265-dcd53bdbc95f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:42:19.453735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453588    4891 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-credential-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:42:19.453735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453649    4891 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-socket\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:42:19.453735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453667    4891 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-workload-certs\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:42:19.453735 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453697    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-data" pod "f1cdffd6-0920-40c6-b265-dcd53bdbc95f" (UID: "f1cdffd6-0920-40c6-b265-dcd53bdbc95f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:42:19.453983 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.453820    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istiod-ca-cert" pod "f1cdffd6-0920-40c6-b265-dcd53bdbc95f" (UID: "f1cdffd6-0920-40c6-b265-dcd53bdbc95f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:42:19.455479 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.455446    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-envoy" pod "f1cdffd6-0920-40c6-b265-dcd53bdbc95f" (UID: "f1cdffd6-0920-40c6-b265-dcd53bdbc95f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:42:19.455583 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.455452    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-podinfo" pod "f1cdffd6-0920-40c6-b265-dcd53bdbc95f" (UID: "f1cdffd6-0920-40c6-b265-dcd53bdbc95f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:42:19.455730 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.455710    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-token" pod "f1cdffd6-0920-40c6-b265-dcd53bdbc95f" (UID: "f1cdffd6-0920-40c6-b265-dcd53bdbc95f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:42:19.455798 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.455729    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-kube-api-access-ztrjj" pod "f1cdffd6-0920-40c6-b265-dcd53bdbc95f" (UID: "f1cdffd6-0920-40c6-b265-dcd53bdbc95f"). InnerVolumeSpecName "kube-api-access-ztrjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:42:19.554327 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.554285    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztrjj\" (UniqueName: \"kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-kube-api-access-ztrjj\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:42:19.554712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.554338    4891 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-token\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:42:19.554712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.554355    4891 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-podinfo\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:42:19.554712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.554371    4891 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istiod-ca-cert\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:42:19.554712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.554380    4891 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-envoy\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:42:19.554712 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.554388    4891 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1cdffd6-0920-40c6-b265-dcd53bdbc95f-istio-data\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 15:42:19.795000 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.794969    4891 generic.go:356] "Generic (PLEG): container finished" podID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerID="7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c" exitCode=0
Nov 02 15:42:19.795197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.795027    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"
Nov 02 15:42:19.795197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.795060    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" event={"ID":"f1cdffd6-0920-40c6-b265-dcd53bdbc95f","Type":"ContainerDied","Data":"7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c"}
Nov 02 15:42:19.795197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.795129    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx" event={"ID":"f1cdffd6-0920-40c6-b265-dcd53bdbc95f","Type":"ContainerDied","Data":"66cca988d162276414aff946f72c0db47b9a8d47f0e0ba250064e501b0c5840b"}
Nov 02 15:42:19.795197 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.795161    4891 scope.go:122] "RemoveContainer" containerID="7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c"
Nov 02 15:42:19.807250 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.807229    4891 scope.go:122] "RemoveContainer" containerID="7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c"
Nov 02 15:42:19.807577 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:42:19.807559    4891 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c\": container with ID starting with 7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c not found: ID does not exist" containerID="7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c"
Nov 02 15:42:19.807665 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.807586    4891 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c"} err="failed to get container status \"7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c\": rpc error: code = NotFound desc = could not find container \"7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c\": container with ID starting with 7846a8e839f267d26ce39f5590a7bc00b4f1a2c042674865f03772a112ff2c3c not found: ID does not exist"
Nov 02 15:42:19.961213 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:19.961182    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"]
Nov 02 15:42:20.010634 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:20.010601    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-019efe-openshift-default-5d87cb7859-dzdcx"]
Nov 02 15:42:20.521150 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:20.521117    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" path="/var/lib/kubelet/pods/f1cdffd6-0920-40c6-b265-dcd53bdbc95f/volumes"
Nov 02 15:42:35.810403 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:35.810366    4891 scope.go:122] "RemoveContainer" containerID="e1f489f1413e998221c28aba3a92695cf36e8ddd57a430831e25b6a2d46027f5"
Nov 02 15:42:44.272872 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:42:44.272826    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:42:59.051606 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:59.051569    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"]
Nov 02 15:42:59.101188 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:42:59.101151    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c064032d-bflnc"]
Nov 02 15:43:00.520349 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:43:00.520299    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7edbba32-9cdc-4bcb-ae47-aabfb8278f6d" path="/var/lib/kubelet/pods/7edbba32-9cdc-4bcb-ae47-aabfb8278f6d/volumes"
Nov 02 15:43:14.273126 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:43:14.273076    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:43:35.845870 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:43:35.845836    4891 scope.go:122] "RemoveContainer" containerID="04b8a1578a5fa6ffea94a6b265964ba1b5108f13896fbc945fea3778db251119"
Nov 02 15:43:35.855009 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:43:35.854988    4891 scope.go:122] "RemoveContainer" containerID="98974306c7bb4cfa666d5449a30a52f332318f373cac3b10b1882a75b679f120"
Nov 02 15:43:35.863307 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:43:35.863289    4891 scope.go:122] "RemoveContainer" containerID="9173ce2d0e34a28a84ea0ad6da67fabf47832cc5db343133481c6556dfdc271c"
Nov 02 15:43:44.273286 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:43:44.273207    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:44:14.272888 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:44:14.272853    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:44:35.892515 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:44:35.892481    4891 scope.go:122] "RemoveContainer" containerID="7351e6bcd4c28bdd2d09e82eed2aa4d9f2abc639c539d26b6a90d969fb35d0fe"
Nov 02 15:44:44.273812 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:44:44.273769    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:45:14.274125 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:45:14.274042    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:45:44.273589 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:45:44.273551    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:46:14.272810 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:46:14.272773    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:46:33.325299 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:46:33.325269    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:46:33.326362 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:46:33.326339    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:46:33.329867 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:46:33.329847    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:46:33.330934 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:46:33.330917    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:46:33.340447 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:46:33.340430    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:46:33.345263 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:46:33.345243    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:46:35.925302 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:46:35.925264    4891 scope.go:122] "RemoveContainer" containerID="322475fe9bb3cb47f25dd52aecf39737e1a64b8fdb369cb51aca0293015e6647"
Nov 02 15:46:44.272975 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:46:44.272886    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:46:53.072460 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:46:53.072422    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"]
Nov 02 15:46:53.121915 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:46:53.121877    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b447c72a-ctpkm"]
Nov 02 15:46:54.520897 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:46:54.520855    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf48e4cf-9553-45d6-8213-3bc9b614b631" path="/var/lib/kubelet/pods/cf48e4cf-9553-45d6-8213-3bc9b614b631/volumes"
Nov 02 15:47:14.272504 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:47:14.272469    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:47:35.947157 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:47:35.947123    4891 scope.go:122] "RemoveContainer" containerID="b67b4205265ccc0e793a2e5f79cbc33be7c7812ee5a5b3374cf4fcb5c322dcd0"
Nov 02 15:47:35.956321 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:47:35.956120    4891 scope.go:122] "RemoveContainer" containerID="402efe581ba2894ca98663d66d70c08384e173fd75436f6b4609277f8497ae4a"
Nov 02 15:47:44.273504 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:47:44.273467    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:48:12.977520 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:48:12.977420    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:48:12.978432 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:48:12.978393    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-j8c7c" podUID="4ecc4b7a-342a-4d11-ae0d-6731df9030aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:48:13.160518 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:48:13.160453    4891 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:48:13.160984 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:48:13.160948    4891 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-gv6hc" podUID="dc99f35f-221e-44ab-bd39-87d52079f398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:48:14.274189 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:48:14.274080    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:48:35.981120 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:48:35.981085    4891 scope.go:122] "RemoveContainer" containerID="8c4af0f1b404e86249850755662b949ebba6e6e885c55ea36cb6e2f11b0a656c"
Nov 02 15:48:35.989880 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:48:35.989861    4891 scope.go:122] "RemoveContainer" containerID="f09979907bab6cbc567b1822706c720a9c54c247cbc0dba50771b4efc993de80"
Nov 02 15:48:44.273365 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:48:44.273302    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:49:14.273585 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:49:14.273547    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:49:44.272966 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:49:44.272880    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:50:14.273330 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:50:14.273278    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:50:44.274110 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:50:44.274059    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:51:14.273074 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:51:14.272994    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:51:33.363452 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:51:33.363420    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:51:33.364561 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:51:33.364543    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:51:33.375008 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:51:33.374982    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:51:33.376043 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:51:33.376019    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:51:33.383474 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:51:33.383454    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:51:33.390264 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:51:33.390247    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:51:44.275102 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:51:44.275064    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:52:14.273793 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:52:14.273752    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:52:44.273430 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:52:44.273330    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:53:14.272451 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:53:14.272415    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:53:44.272410 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:53:44.272366    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:54:14.273212 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:54:14.273124    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:54:44.274046 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:54:44.274008    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:55:14.272804 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:55:14.272768    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:55:44.273568 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:55:44.273484    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:56:14.273775 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:56:14.273733    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:56:33.407137 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:56:33.407104    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:56:33.408225 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:56:33.408201    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:56:33.415774 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:56:33.415750    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 15:56:33.416806 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:56:33.416787    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 15:56:33.422780 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:56:33.422760    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:56:33.431114 ip-10-0-132-27 kubenswrapper[4891]: I1102 15:56:33.431096    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 15:56:44.273425 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:56:44.273387    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:57:14.273774 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:57:14.273690    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:57:44.273647 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:57:44.273610    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:58:14.274059 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:58:14.274016    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:58:44.273462 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:58:44.273384    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:59:14.273793 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:59:14.273757    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 15:59:44.273071 ip-10-0-132-27 kubenswrapper[4891]: E1102 15:59:44.273034    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:00:00.169944 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.169911    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"]
Nov 02 16:00:00.170455 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.170291    4891 cpu_manager.go:414] "containerMap: removing container" podUID="cf48e4cf-9553-45d6-8213-3bc9b614b631" containerName="revoke-keys"
Nov 02 16:00:00.170455 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.170302    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf48e4cf-9553-45d6-8213-3bc9b614b631" containerName="revoke-keys"
Nov 02 16:00:00.170455 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.170332    4891 cpu_manager.go:414] "containerMap: removing container" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerName="istio-proxy"
Nov 02 16:00:00.170455 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.170338    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerName="istio-proxy"
Nov 02 16:00:00.170455 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.170361    4891 cpu_manager.go:414] "containerMap: removing container" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main"
Nov 02 16:00:00.170455 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.170366    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main"
Nov 02 16:00:00.170455 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.170382    4891 cpu_manager.go:414] "containerMap: removing container" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main"
Nov 02 16:00:00.170455 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.170386    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main"
Nov 02 16:00:00.170455 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.170446    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="4da5799a-02da-4d12-98ae-7eb568707007" containerName="main"
Nov 02 16:00:00.170455 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.170456    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4f9a33-e005-4bd4-97a4-e65d83ef4220" containerName="main"
Nov 02 16:00:00.170781 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.170464    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf48e4cf-9553-45d6-8213-3bc9b614b631" containerName="revoke-keys"
Nov 02 16:00:00.170781 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.170470    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cdffd6-0920-40c6-b265-dcd53bdbc95f" containerName="istio-proxy"
Nov 02 16:00:00.173564 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.173547    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"
Nov 02 16:00:00.232352 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.232304    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"]
Nov 02 16:00:00.278641 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.278609    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:00:00.278641 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.278608    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:00:00.278866 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.278657    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6pzqc\""
Nov 02 16:00:00.330435 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.330401    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c73b97b-7bf8-43b7-a697-f34c7a25d579-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893920-p6kj7\" (UID: \"5c73b97b-7bf8-43b7-a697-f34c7a25d579\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"
Nov 02 16:00:00.330609 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.330439    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4p9h\" (UniqueName: \"kubernetes.io/projected/5c73b97b-7bf8-43b7-a697-f34c7a25d579-kube-api-access-f4p9h\") pod \"maas-api-key-cleanup-29893920-p6kj7\" (UID: \"5c73b97b-7bf8-43b7-a697-f34c7a25d579\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"
Nov 02 16:00:00.431386 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.431269    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4p9h\" (UniqueName: \"kubernetes.io/projected/5c73b97b-7bf8-43b7-a697-f34c7a25d579-kube-api-access-f4p9h\") pod \"maas-api-key-cleanup-29893920-p6kj7\" (UID: \"5c73b97b-7bf8-43b7-a697-f34c7a25d579\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"
Nov 02 16:00:00.431592 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.431446    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c73b97b-7bf8-43b7-a697-f34c7a25d579-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893920-p6kj7\" (UID: \"5c73b97b-7bf8-43b7-a697-f34c7a25d579\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"
Nov 02 16:00:00.432100 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.432079    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c73b97b-7bf8-43b7-a697-f34c7a25d579-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893920-p6kj7\" (UID: \"5c73b97b-7bf8-43b7-a697-f34c7a25d579\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"
Nov 02 16:00:00.493029 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.492995    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4p9h\" (UniqueName: \"kubernetes.io/projected/5c73b97b-7bf8-43b7-a697-f34c7a25d579-kube-api-access-f4p9h\") pod \"maas-api-key-cleanup-29893920-p6kj7\" (UID: \"5c73b97b-7bf8-43b7-a697-f34c7a25d579\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"
Nov 02 16:00:00.790268 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:00.790240    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"
Nov 02 16:00:01.165092 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:01.165064    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"]
Nov 02 16:00:01.167383 ip-10-0-132-27 kubenswrapper[4891]: W1102 16:00:01.167351    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c73b97b_7bf8_43b7_a697_f34c7a25d579.slice/crio-90f5ad58e8ee726505e898708d3fcf3cd9752f6fd9fc93f802defb265d1bae5b WatchSource:0}: Error finding container 90f5ad58e8ee726505e898708d3fcf3cd9752f6fd9fc93f802defb265d1bae5b: Status 404 returned error can't find the container with id 90f5ad58e8ee726505e898708d3fcf3cd9752f6fd9fc93f802defb265d1bae5b
Nov 02 16:00:01.282982 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:01.282945    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7" event={"ID":"5c73b97b-7bf8-43b7-a697-f34c7a25d579","Type":"ContainerStarted","Data":"90f5ad58e8ee726505e898708d3fcf3cd9752f6fd9fc93f802defb265d1bae5b"}
Nov 02 16:00:02.290056 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:02.290014    4891 generic.go:356] "Generic (PLEG): container finished" podID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" containerID="7be9b1a765c2dc418eb4e5ef842a0f95b284d847fa42386773e83ffca9495dba" exitCode=60
Nov 02 16:00:02.290548 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:02.290103    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7" event={"ID":"5c73b97b-7bf8-43b7-a697-f34c7a25d579","Type":"ContainerDied","Data":"7be9b1a765c2dc418eb4e5ef842a0f95b284d847fa42386773e83ffca9495dba"}
Nov 02 16:00:02.290548 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:02.290300    4891 scope.go:122] "RemoveContainer" containerID="7be9b1a765c2dc418eb4e5ef842a0f95b284d847fa42386773e83ffca9495dba"
Nov 02 16:00:03.296811 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:03.296777    4891 generic.go:356] "Generic (PLEG): container finished" podID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" containerID="d33f19b03c0f9e9bef84a8acf8712b34e63424aef7b8d8c9c6975e5d090162db" exitCode=60
Nov 02 16:00:03.297303 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:03.296848    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7" event={"ID":"5c73b97b-7bf8-43b7-a697-f34c7a25d579","Type":"ContainerDied","Data":"d33f19b03c0f9e9bef84a8acf8712b34e63424aef7b8d8c9c6975e5d090162db"}
Nov 02 16:00:03.297303 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:03.296891    4891 scope.go:122] "RemoveContainer" containerID="7be9b1a765c2dc418eb4e5ef842a0f95b284d847fa42386773e83ffca9495dba"
Nov 02 16:00:03.297303 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:03.297210    4891 scope.go:122] "RemoveContainer" containerID="d33f19b03c0f9e9bef84a8acf8712b34e63424aef7b8d8c9c6975e5d090162db"
Nov 02 16:00:03.297552 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:00:03.297527    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893920-p6kj7_odh-ai-gateway-infra(5c73b97b-7bf8-43b7-a697-f34c7a25d579)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579"
Nov 02 16:00:04.304189 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:04.304157    4891 scope.go:122] "RemoveContainer" containerID="d33f19b03c0f9e9bef84a8acf8712b34e63424aef7b8d8c9c6975e5d090162db"
Nov 02 16:00:04.304630 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:00:04.304416    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893920-p6kj7_odh-ai-gateway-infra(5c73b97b-7bf8-43b7-a697-f34c7a25d579)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579"
Nov 02 16:00:14.273633 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:00:14.273548    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:00:16.514841 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:16.514809    4891 scope.go:122] "RemoveContainer" containerID="d33f19b03c0f9e9bef84a8acf8712b34e63424aef7b8d8c9c6975e5d090162db"
Nov 02 16:00:17.376419 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:17.376390    4891 generic.go:356] "Generic (PLEG): container finished" podID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" containerID="ab2d979667efab3b56f028171f12a5d0cad6c04e9d00617cd08364566825e99f" exitCode=60
Nov 02 16:00:17.376567 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:17.376470    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7" event={"ID":"5c73b97b-7bf8-43b7-a697-f34c7a25d579","Type":"ContainerDied","Data":"ab2d979667efab3b56f028171f12a5d0cad6c04e9d00617cd08364566825e99f"}
Nov 02 16:00:17.376567 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:17.376533    4891 scope.go:122] "RemoveContainer" containerID="d33f19b03c0f9e9bef84a8acf8712b34e63424aef7b8d8c9c6975e5d090162db"
Nov 02 16:00:17.376953 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:17.376936    4891 scope.go:122] "RemoveContainer" containerID="ab2d979667efab3b56f028171f12a5d0cad6c04e9d00617cd08364566825e99f"
Nov 02 16:00:17.377229 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:00:17.377207    4891 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893920-p6kj7_odh-ai-gateway-infra(5c73b97b-7bf8-43b7-a697-f34c7a25d579)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579"
Nov 02 16:00:18.505644 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:18.505605    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"]
Nov 02 16:00:18.646552 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:18.646526    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"
Nov 02 16:00:18.704142 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:18.704106    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c73b97b-7bf8-43b7-a697-f34c7a25d579-kube-api-access-f4p9h\" (UniqueName: \"kubernetes.io/projected/5c73b97b-7bf8-43b7-a697-f34c7a25d579-kube-api-access-f4p9h\") pod \"5c73b97b-7bf8-43b7-a697-f34c7a25d579\" (UID: \"5c73b97b-7bf8-43b7-a697-f34c7a25d579\") "
Nov 02 16:00:18.704383 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:18.704360    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c73b97b-7bf8-43b7-a697-f34c7a25d579-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c73b97b-7bf8-43b7-a697-f34c7a25d579-maas-api-service-ca\") pod \"5c73b97b-7bf8-43b7-a697-f34c7a25d579\" (UID: \"5c73b97b-7bf8-43b7-a697-f34c7a25d579\") "
Nov 02 16:00:18.704716 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:18.704692    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c73b97b-7bf8-43b7-a697-f34c7a25d579-maas-api-service-ca" pod "5c73b97b-7bf8-43b7-a697-f34c7a25d579" (UID: "5c73b97b-7bf8-43b7-a697-f34c7a25d579"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:00:18.706281 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:18.706256    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c73b97b-7bf8-43b7-a697-f34c7a25d579-kube-api-access-f4p9h" pod "5c73b97b-7bf8-43b7-a697-f34c7a25d579" (UID: "5c73b97b-7bf8-43b7-a697-f34c7a25d579"). InnerVolumeSpecName "kube-api-access-f4p9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:00:18.806055 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:18.806014    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c73b97b-7bf8-43b7-a697-f34c7a25d579-maas-api-service-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:00:18.806228 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:18.806059    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4p9h\" (UniqueName: \"kubernetes.io/projected/5c73b97b-7bf8-43b7-a697-f34c7a25d579-kube-api-access-f4p9h\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:00:19.390752 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:19.390718    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"
Nov 02 16:00:19.390940 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:19.390749    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7" event={"ID":"5c73b97b-7bf8-43b7-a697-f34c7a25d579","Type":"ContainerDied","Data":"90f5ad58e8ee726505e898708d3fcf3cd9752f6fd9fc93f802defb265d1bae5b"}
Nov 02 16:00:19.390940 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:19.390812    4891 scope.go:122] "RemoveContainer" containerID="ab2d979667efab3b56f028171f12a5d0cad6c04e9d00617cd08364566825e99f"
Nov 02 16:00:19.554987 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:19.554950    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"]
Nov 02 16:00:19.604420 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:19.604384    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-p6kj7"]
Nov 02 16:00:20.527230 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:00:20.521388    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" path="/var/lib/kubelet/pods/5c73b97b-7bf8-43b7-a697-f34c7a25d579/volumes"
Nov 02 16:00:44.273746 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:00:44.273709    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:01:06.382583 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.382545    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"]
Nov 02 16:01:06.383000 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.382940    4891 cpu_manager.go:414] "containerMap: removing container" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" containerName="cleanup"
Nov 02 16:01:06.383000 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.382951    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" containerName="cleanup"
Nov 02 16:01:06.383000 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.382964    4891 cpu_manager.go:414] "containerMap: removing container" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" containerName="cleanup"
Nov 02 16:01:06.383000 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.382969    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" containerName="cleanup"
Nov 02 16:01:06.383000 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.382993    4891 cpu_manager.go:414] "containerMap: removing container" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" containerName="cleanup"
Nov 02 16:01:06.383000 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.382999    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" containerName="cleanup"
Nov 02 16:01:06.383251 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.383060    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" containerName="cleanup"
Nov 02 16:01:06.383251 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.383070    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" containerName="cleanup"
Nov 02 16:01:06.386113 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.386097    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:06.433287 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.433245    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82vn4\" (UniqueName: \"kubernetes.io/projected/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-kube-api-access-82vn4\") pod \"maas-api-revoke-keys-e2e-ait-5315ece5-hdddv\" (UID: \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:06.433523 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.433298    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5315ece5-hdddv\" (UID: \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:06.433523 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.433446    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5315ece5-hdddv\" (UID: \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:06.440935 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.440903    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:01:06.490447 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.490407    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:01:06.490447 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.490413    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:01:06.490674 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.490472    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7brzf\""
Nov 02 16:01:06.492863 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.492842    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"]
Nov 02 16:01:06.534834 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.534802    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82vn4\" (UniqueName: \"kubernetes.io/projected/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-kube-api-access-82vn4\") pod \"maas-api-revoke-keys-e2e-ait-5315ece5-hdddv\" (UID: \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:06.535042 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.534851    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5315ece5-hdddv\" (UID: \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:06.535042 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.534920    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5315ece5-hdddv\" (UID: \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:06.535610 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.535587    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5315ece5-hdddv\" (UID: \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:06.537303 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.537283    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5315ece5-hdddv\" (UID: \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:06.596318 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.596276    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82vn4\" (UniqueName: \"kubernetes.io/projected/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-kube-api-access-82vn4\") pod \"maas-api-revoke-keys-e2e-ait-5315ece5-hdddv\" (UID: \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:06.698929 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:06.698852    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:07.075773 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:07.075750    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"]
Nov 02 16:01:07.078030 ip-10-0-132-27 kubenswrapper[4891]: W1102 16:01:07.077995    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f3e7b6_04a2_4cfa_a295_82c5b9cff1e1.slice/crio-23f15cdf4c887ef57e390c8965f79c0b30592273558b2f8242e2fe4dace3c28c WatchSource:0}: Error finding container 23f15cdf4c887ef57e390c8965f79c0b30592273558b2f8242e2fe4dace3c28c: Status 404 returned error can't find the container with id 23f15cdf4c887ef57e390c8965f79c0b30592273558b2f8242e2fe4dace3c28c
Nov 02 16:01:07.650660 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:07.650566    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv" event={"ID":"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1","Type":"ContainerStarted","Data":"1f5d0133e0f529e4f2e6f00b2948bdc5a2077538653848e14e4847b30eaa31cd"}
Nov 02 16:01:07.651100 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:07.650661    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv" event={"ID":"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1","Type":"ContainerStarted","Data":"23f15cdf4c887ef57e390c8965f79c0b30592273558b2f8242e2fe4dace3c28c"}
Nov 02 16:01:07.768467 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:07.768409    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv" podStartSLOduration=1.76839476 podStartE2EDuration="1.76839476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:01:07.765822568 +0000 UTC m=+3875.358439368" watchObservedRunningTime="2026-11-02 16:01:07.76839476 +0000 UTC m=+3875.361011585"
Nov 02 16:01:08.657397 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:08.657363    4891 generic.go:356] "Generic (PLEG): container finished" podID="95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1" containerID="1f5d0133e0f529e4f2e6f00b2948bdc5a2077538653848e14e4847b30eaa31cd" exitCode=0
Nov 02 16:01:08.657809 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:08.657450    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv" event={"ID":"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1","Type":"ContainerDied","Data":"1f5d0133e0f529e4f2e6f00b2948bdc5a2077538653848e14e4847b30eaa31cd"}
Nov 02 16:01:09.794231 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:09.794202    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:09.865050 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:09.865016    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-kube-api-access-82vn4\" (UniqueName: \"kubernetes.io/projected/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-kube-api-access-82vn4\") pod \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\" (UID: \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\") "
Nov 02 16:01:09.865246 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:09.865076    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-xks-ca\") pod \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\" (UID: \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\") "
Nov 02 16:01:09.865246 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:09.865118    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-openshift-ca\") pod \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\" (UID: \"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1\") "
Nov 02 16:01:09.865588 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:09.865564    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-openshift-ca" pod "95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1" (UID: "95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:01:09.867289 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:09.867266    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-kube-api-access-82vn4" pod "95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1" (UID: "95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1"). InnerVolumeSpecName "kube-api-access-82vn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:01:09.867413 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:09.867287    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-xks-ca" pod "95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1" (UID: "95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:01:09.966162 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:09.966085    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:01:09.966162 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:09.966114    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82vn4\" (UniqueName: \"kubernetes.io/projected/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-kube-api-access-82vn4\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:01:09.966162 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:09.966126    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:01:10.670247 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:10.670213    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"
Nov 02 16:01:10.670452 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:10.670212    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv" event={"ID":"95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1","Type":"ContainerDied","Data":"23f15cdf4c887ef57e390c8965f79c0b30592273558b2f8242e2fe4dace3c28c"}
Nov 02 16:01:10.670452 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:10.670372    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23f15cdf4c887ef57e390c8965f79c0b30592273558b2f8242e2fe4dace3c28c"
Nov 02 16:01:14.274382 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:01:14.274348    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:01:22.561416 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:22.561383    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"]
Nov 02 16:01:22.610809 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:22.610776    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5315ece5-hdddv"]
Nov 02 16:01:24.520247 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:24.520214    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1" path="/var/lib/kubelet/pods/95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1/volumes"
Nov 02 16:01:33.446203 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:33.446175    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 16:01:33.447273 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:33.447249    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 16:01:33.456000 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:33.455979    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 16:01:33.457049 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:33.457028    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 16:01:33.463148 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:33.463129    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 16:01:33.473098 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:01:33.473077    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 16:01:44.276196 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:01:44.276122    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:02:14.273101 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:02:14.273063    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:02:44.274123 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:02:44.274077    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:03:14.273177 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:03:14.273096    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:03:20.012519 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.012478    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"]
Nov 02 16:03:20.012946 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.012900    4891 cpu_manager.go:414] "containerMap: removing container" podUID="95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1" containerName="revoke-keys"
Nov 02 16:03:20.012946 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.012911    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1" containerName="revoke-keys"
Nov 02 16:03:20.013040 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.012974    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c73b97b-7bf8-43b7-a697-f34c7a25d579" containerName="cleanup"
Nov 02 16:03:20.013040 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.012988    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f3e7b6-04a2-4cfa-a295-82c5b9cff1e1" containerName="revoke-keys"
Nov 02 16:03:20.015998 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.015980    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:20.120353 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.120296    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:03:20.120353 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.120335    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:03:20.120353 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.120296    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:03:20.120614 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.120410    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7brzf\""
Nov 02 16:03:20.122931 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.122908    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"]
Nov 02 16:03:20.167499 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.167464    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scn4r\" (UniqueName: \"kubernetes.io/projected/da9e8575-6ea1-43cb-8338-07ceef27a03f-kube-api-access-scn4r\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6\" (UID: \"da9e8575-6ea1-43cb-8338-07ceef27a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:20.167667 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.167511    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6\" (UID: \"da9e8575-6ea1-43cb-8338-07ceef27a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:20.167667 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.167651    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6\" (UID: \"da9e8575-6ea1-43cb-8338-07ceef27a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:20.269165 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.269130    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6\" (UID: \"da9e8575-6ea1-43cb-8338-07ceef27a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:20.269397 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.269267    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scn4r\" (UniqueName: \"kubernetes.io/projected/da9e8575-6ea1-43cb-8338-07ceef27a03f-kube-api-access-scn4r\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6\" (UID: \"da9e8575-6ea1-43cb-8338-07ceef27a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:20.269397 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.269307    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6\" (UID: \"da9e8575-6ea1-43cb-8338-07ceef27a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:20.269993 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.269965    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6\" (UID: \"da9e8575-6ea1-43cb-8338-07ceef27a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:20.271534 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.271513    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6\" (UID: \"da9e8575-6ea1-43cb-8338-07ceef27a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:20.330442 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.330410    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scn4r\" (UniqueName: \"kubernetes.io/projected/da9e8575-6ea1-43cb-8338-07ceef27a03f-kube-api-access-scn4r\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6\" (UID: \"da9e8575-6ea1-43cb-8338-07ceef27a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:20.629519 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:20.629434    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:21.005955 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:21.005926    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"]
Nov 02 16:03:21.007664 ip-10-0-132-27 kubenswrapper[4891]: W1102 16:03:21.007635    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9e8575_6ea1_43cb_8338_07ceef27a03f.slice/crio-4bacecbe82910665ac27993093484bf768836b815dbbea668eefa00d5567dbd1 WatchSource:0}: Error finding container 4bacecbe82910665ac27993093484bf768836b815dbbea668eefa00d5567dbd1: Status 404 returned error can't find the container with id 4bacecbe82910665ac27993093484bf768836b815dbbea668eefa00d5567dbd1
Nov 02 16:03:21.380296 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:21.380256    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6" event={"ID":"da9e8575-6ea1-43cb-8338-07ceef27a03f","Type":"ContainerStarted","Data":"6ef248735b994deb51dd1b6e7e99048a88fe4847bc850f998649778d21cd13e6"}
Nov 02 16:03:21.380756 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:21.380345    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6" event={"ID":"da9e8575-6ea1-43cb-8338-07ceef27a03f","Type":"ContainerStarted","Data":"4bacecbe82910665ac27993093484bf768836b815dbbea668eefa00d5567dbd1"}
Nov 02 16:03:21.497101 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:21.497050    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6" podStartSLOduration=2.497036356 podStartE2EDuration="2.497036356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:03:21.495665031 +0000 UTC m=+4009.088281832" watchObservedRunningTime="2026-11-02 16:03:21.497036356 +0000 UTC m=+4009.089653149"
Nov 02 16:03:22.386956 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:22.386920    4891 generic.go:356] "Generic (PLEG): container finished" podID="da9e8575-6ea1-43cb-8338-07ceef27a03f" containerID="6ef248735b994deb51dd1b6e7e99048a88fe4847bc850f998649778d21cd13e6" exitCode=0
Nov 02 16:03:22.387387 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:22.387014    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6" event={"ID":"da9e8575-6ea1-43cb-8338-07ceef27a03f","Type":"ContainerDied","Data":"6ef248735b994deb51dd1b6e7e99048a88fe4847bc850f998649778d21cd13e6"}
Nov 02 16:03:23.538927 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:23.538895    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:23.703626 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:23.703525    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9e8575-6ea1-43cb-8338-07ceef27a03f-kube-api-access-scn4r\" (UniqueName: \"kubernetes.io/projected/da9e8575-6ea1-43cb-8338-07ceef27a03f-kube-api-access-scn4r\") pod \"da9e8575-6ea1-43cb-8338-07ceef27a03f\" (UID: \"da9e8575-6ea1-43cb-8338-07ceef27a03f\") "
Nov 02 16:03:23.703841 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:23.703640    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-openshift-ca\") pod \"da9e8575-6ea1-43cb-8338-07ceef27a03f\" (UID: \"da9e8575-6ea1-43cb-8338-07ceef27a03f\") "
Nov 02 16:03:23.703841 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:23.703676    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-xks-ca\") pod \"da9e8575-6ea1-43cb-8338-07ceef27a03f\" (UID: \"da9e8575-6ea1-43cb-8338-07ceef27a03f\") "
Nov 02 16:03:23.704071 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:23.704050    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-openshift-ca" pod "da9e8575-6ea1-43cb-8338-07ceef27a03f" (UID: "da9e8575-6ea1-43cb-8338-07ceef27a03f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:03:23.705711 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:23.705686    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9e8575-6ea1-43cb-8338-07ceef27a03f-kube-api-access-scn4r" pod "da9e8575-6ea1-43cb-8338-07ceef27a03f" (UID: "da9e8575-6ea1-43cb-8338-07ceef27a03f"). InnerVolumeSpecName "kube-api-access-scn4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:03:23.705814 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:23.705705    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-xks-ca" pod "da9e8575-6ea1-43cb-8338-07ceef27a03f" (UID: "da9e8575-6ea1-43cb-8338-07ceef27a03f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:03:23.805002 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:23.804966    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:03:23.805002 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:23.804998    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9e8575-6ea1-43cb-8338-07ceef27a03f-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:03:23.805002 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:23.805008    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scn4r\" (UniqueName: \"kubernetes.io/projected/da9e8575-6ea1-43cb-8338-07ceef27a03f-kube-api-access-scn4r\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:03:24.399476 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:24.399439    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6" event={"ID":"da9e8575-6ea1-43cb-8338-07ceef27a03f","Type":"ContainerDied","Data":"4bacecbe82910665ac27993093484bf768836b815dbbea668eefa00d5567dbd1"}
Nov 02 16:03:24.399476 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:24.399462    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"
Nov 02 16:03:24.399696 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:24.399496    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bacecbe82910665ac27993093484bf768836b815dbbea668eefa00d5567dbd1"
Nov 02 16:03:43.063926 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:43.063889    4891 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"]
Nov 02 16:03:43.112888 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:43.112853    4891 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2afef9-89gz6"]
Nov 02 16:03:44.273989 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:03:44.273950    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:03:44.527018 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:03:44.526931    4891 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da9e8575-6ea1-43cb-8338-07ceef27a03f" path="/var/lib/kubelet/pods/da9e8575-6ea1-43cb-8338-07ceef27a03f/volumes"
Nov 02 16:04:14.273348 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:04:14.273288    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:04:17.706183 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.706146    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"]
Nov 02 16:04:17.706626 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.706611    4891 cpu_manager.go:414] "containerMap: removing container" podUID="da9e8575-6ea1-43cb-8338-07ceef27a03f" containerName="revoke-keys"
Nov 02 16:04:17.706686 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.706628    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9e8575-6ea1-43cb-8338-07ceef27a03f" containerName="revoke-keys"
Nov 02 16:04:17.706724 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.706699    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9e8575-6ea1-43cb-8338-07ceef27a03f" containerName="revoke-keys"
Nov 02 16:04:17.709817 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.709802    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:17.770746 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.770710    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-gjx87\" (UID: \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:17.770924 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.770757    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdthf\" (UniqueName: \"kubernetes.io/projected/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-kube-api-access-kdthf\") pod \"maas-api-revoke-sub-f03629db-gjx87\" (UID: \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:17.770924 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.770808    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-gjx87\" (UID: \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:17.813818 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.813785    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:04:17.813818 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.813784    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:04:17.813818 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.813815    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:04:17.814078 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.813956    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7brzf\""
Nov 02 16:04:17.816270 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.816248    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"]
Nov 02 16:04:17.871873 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.871831    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-gjx87\" (UID: \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:17.871873 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.871873    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdthf\" (UniqueName: \"kubernetes.io/projected/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-kube-api-access-kdthf\") pod \"maas-api-revoke-sub-f03629db-gjx87\" (UID: \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:17.872103 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.872062    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-gjx87\" (UID: \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:17.872754 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.872728    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-gjx87\" (UID: \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:17.874359 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.874337    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-gjx87\" (UID: \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:17.932605 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:17.932567    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdthf\" (UniqueName: \"kubernetes.io/projected/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-kube-api-access-kdthf\") pod \"maas-api-revoke-sub-f03629db-gjx87\" (UID: \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:18.022503 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:18.022477    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:18.395637 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:18.395608    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"]
Nov 02 16:04:18.398057 ip-10-0-132-27 kubenswrapper[4891]: W1102 16:04:18.398028    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cf2cd4_c831_427e_9fd1_b86357bd3e5b.slice/crio-bec80a1d146316bf19d92119a5db896ecc980521882ef7ed131fdee4b174e8f2 WatchSource:0}: Error finding container bec80a1d146316bf19d92119a5db896ecc980521882ef7ed131fdee4b174e8f2: Status 404 returned error can't find the container with id bec80a1d146316bf19d92119a5db896ecc980521882ef7ed131fdee4b174e8f2
Nov 02 16:04:18.746389 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:18.746348    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87" event={"ID":"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b","Type":"ContainerStarted","Data":"80222955aab9d0af0b2c8ef846fd0fcbc40330e551956d5b4c70da7a09261352"}
Nov 02 16:04:18.746885 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:18.746416    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87" event={"ID":"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b","Type":"ContainerStarted","Data":"bec80a1d146316bf19d92119a5db896ecc980521882ef7ed131fdee4b174e8f2"}
Nov 02 16:04:18.864075 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:18.864015    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87" podStartSLOduration=1.8639988440000002 podStartE2EDuration="1.863998844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:04:18.861696962 +0000 UTC m=+4066.454313763" watchObservedRunningTime="2026-11-02 16:04:18.863998844 +0000 UTC m=+4066.456615649"
Nov 02 16:04:19.753573 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:19.753538    4891 generic.go:356] "Generic (PLEG): container finished" podID="a0cf2cd4-c831-427e-9fd1-b86357bd3e5b" containerID="80222955aab9d0af0b2c8ef846fd0fcbc40330e551956d5b4c70da7a09261352" exitCode=0
Nov 02 16:04:19.753989 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:19.753628    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87" event={"ID":"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b","Type":"ContainerDied","Data":"80222955aab9d0af0b2c8ef846fd0fcbc40330e551956d5b4c70da7a09261352"}
Nov 02 16:04:20.895730 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:20.895698    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:21.000104 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:21.000069    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-xks-ca\") pod \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\" (UID: \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\") "
Nov 02 16:04:21.000273 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:21.000140    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-openshift-ca\") pod \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\" (UID: \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\") "
Nov 02 16:04:21.000273 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:21.000213    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-kube-api-access-kdthf\" (UniqueName: \"kubernetes.io/projected/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-kube-api-access-kdthf\") pod \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\" (UID: \"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b\") "
Nov 02 16:04:21.000629 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:21.000600    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-openshift-ca" pod "a0cf2cd4-c831-427e-9fd1-b86357bd3e5b" (UID: "a0cf2cd4-c831-427e-9fd1-b86357bd3e5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:04:21.002276 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:21.002252    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-xks-ca" pod "a0cf2cd4-c831-427e-9fd1-b86357bd3e5b" (UID: "a0cf2cd4-c831-427e-9fd1-b86357bd3e5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:04:21.002407 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:21.002389    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-kube-api-access-kdthf" pod "a0cf2cd4-c831-427e-9fd1-b86357bd3e5b" (UID: "a0cf2cd4-c831-427e-9fd1-b86357bd3e5b"). InnerVolumeSpecName "kube-api-access-kdthf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:04:21.101299 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:21.101254    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdthf\" (UniqueName: \"kubernetes.io/projected/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-kube-api-access-kdthf\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:04:21.101299 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:21.101293    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:04:21.101299 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:21.101305    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0cf2cd4-c831-427e-9fd1-b86357bd3e5b-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:04:21.765613 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:21.765580    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87"
Nov 02 16:04:21.765802 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:21.765612    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gjx87" event={"ID":"a0cf2cd4-c831-427e-9fd1-b86357bd3e5b","Type":"ContainerDied","Data":"bec80a1d146316bf19d92119a5db896ecc980521882ef7ed131fdee4b174e8f2"}
Nov 02 16:04:21.765802 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:21.765657    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bec80a1d146316bf19d92119a5db896ecc980521882ef7ed131fdee4b174e8f2"
Nov 02 16:04:38.903493 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:38.903456    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"]
Nov 02 16:04:38.904060 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:38.904039    4891 cpu_manager.go:414] "containerMap: removing container" podUID="a0cf2cd4-c831-427e-9fd1-b86357bd3e5b" containerName="revoke-keys"
Nov 02 16:04:38.904156 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:38.904063    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0cf2cd4-c831-427e-9fd1-b86357bd3e5b" containerName="revoke-keys"
Nov 02 16:04:38.904216 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:38.904204    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0cf2cd4-c831-427e-9fd1-b86357bd3e5b" containerName="revoke-keys"
Nov 02 16:04:38.907547 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:38.907524    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:38.962417 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:38.962377    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:04:38.989484 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:38.989446    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:04:38.989654 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:38.989489    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:04:39.011967 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.011933    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7brzf\""
Nov 02 16:04:39.014418 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.014394    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"]
Nov 02 16:04:39.024043 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.024012    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-9m9sp\" (UID: \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:39.024164 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.024059    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-9m9sp\" (UID: \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:39.024303 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.024272    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc9wn\" (UniqueName: \"kubernetes.io/projected/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-kube-api-access-dc9wn\") pod \"maas-api-revoke-sub-0165d7d2-9m9sp\" (UID: \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:39.125844 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.125798    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-9m9sp\" (UID: \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:39.126033 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.125891    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc9wn\" (UniqueName: \"kubernetes.io/projected/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-kube-api-access-dc9wn\") pod \"maas-api-revoke-sub-0165d7d2-9m9sp\" (UID: \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:39.126033 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.125938    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-9m9sp\" (UID: \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:39.126563 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.126537    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-9m9sp\" (UID: \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:39.128241 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.128218    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-9m9sp\" (UID: \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:39.187001 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.186930    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc9wn\" (UniqueName: \"kubernetes.io/projected/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-kube-api-access-dc9wn\") pod \"maas-api-revoke-sub-0165d7d2-9m9sp\" (UID: \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:39.220611 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.220576    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:39.596589 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.596560    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"]
Nov 02 16:04:39.598921 ip-10-0-132-27 kubenswrapper[4891]: W1102 16:04:39.598889    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8a6250_e0bf_4ab4_a3c5_084c93cbc028.slice/crio-65d27a374709df39c446769361f308da2c050182dccc1accf0aee08c6aa145b5 WatchSource:0}: Error finding container 65d27a374709df39c446769361f308da2c050182dccc1accf0aee08c6aa145b5: Status 404 returned error can't find the container with id 65d27a374709df39c446769361f308da2c050182dccc1accf0aee08c6aa145b5
Nov 02 16:04:39.898189 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.898149    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp" event={"ID":"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028","Type":"ContainerStarted","Data":"e171f9699261a8e50a04accafdb680a5d470dc14aeed73c55d4a0769fb23b425"}
Nov 02 16:04:39.898361 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:39.898206    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp" event={"ID":"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028","Type":"ContainerStarted","Data":"65d27a374709df39c446769361f308da2c050182dccc1accf0aee08c6aa145b5"}
Nov 02 16:04:40.020678 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:40.020612    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp" podStartSLOduration=2.020594109 podStartE2EDuration="2.020594109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:04:40.0193687 +0000 UTC m=+4087.611985499" watchObservedRunningTime="2026-11-02 16:04:40.020594109 +0000 UTC m=+4087.613210911"
Nov 02 16:04:40.905215 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:40.905175    4891 generic.go:356] "Generic (PLEG): container finished" podID="6c8a6250-e0bf-4ab4-a3c5-084c93cbc028" containerID="e171f9699261a8e50a04accafdb680a5d470dc14aeed73c55d4a0769fb23b425" exitCode=0
Nov 02 16:04:40.905415 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:40.905259    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp" event={"ID":"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028","Type":"ContainerDied","Data":"e171f9699261a8e50a04accafdb680a5d470dc14aeed73c55d4a0769fb23b425"}
Nov 02 16:04:42.053292 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.053265    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:42.155889 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.155858    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-openshift-ca\") pod \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\" (UID: \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\") "
Nov 02 16:04:42.155889 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.155895    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-xks-ca\") pod \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\" (UID: \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\") "
Nov 02 16:04:42.156167 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.155958    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-kube-api-access-dc9wn\" (UniqueName: \"kubernetes.io/projected/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-kube-api-access-dc9wn\") pod \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\" (UID: \"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028\") "
Nov 02 16:04:42.156253 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.156227    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-openshift-ca" pod "6c8a6250-e0bf-4ab4-a3c5-084c93cbc028" (UID: "6c8a6250-e0bf-4ab4-a3c5-084c93cbc028"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:04:42.158134 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.158107    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-xks-ca" pod "6c8a6250-e0bf-4ab4-a3c5-084c93cbc028" (UID: "6c8a6250-e0bf-4ab4-a3c5-084c93cbc028"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:04:42.158273 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.158149    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-kube-api-access-dc9wn" pod "6c8a6250-e0bf-4ab4-a3c5-084c93cbc028" (UID: "6c8a6250-e0bf-4ab4-a3c5-084c93cbc028"). InnerVolumeSpecName "kube-api-access-dc9wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:04:42.257100 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.257040    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:04:42.257100 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.257093    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:04:42.257100 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.257104    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc9wn\" (UniqueName: \"kubernetes.io/projected/6c8a6250-e0bf-4ab4-a3c5-084c93cbc028-kube-api-access-dc9wn\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:04:42.918536 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.918434    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp"
Nov 02 16:04:42.918536 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.918463    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-9m9sp" event={"ID":"6c8a6250-e0bf-4ab4-a3c5-084c93cbc028","Type":"ContainerDied","Data":"65d27a374709df39c446769361f308da2c050182dccc1accf0aee08c6aa145b5"}
Nov 02 16:04:42.918536 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:42.918513    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65d27a374709df39c446769361f308da2c050182dccc1accf0aee08c6aa145b5"
Nov 02 16:04:44.272737 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:04:44.272697    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:04:50.688936 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.688901    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"]
Nov 02 16:04:50.689476 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.689325    4891 cpu_manager.go:414] "containerMap: removing container" podUID="6c8a6250-e0bf-4ab4-a3c5-084c93cbc028" containerName="revoke-keys"
Nov 02 16:04:50.689476 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.689338    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8a6250-e0bf-4ab4-a3c5-084c93cbc028" containerName="revoke-keys"
Nov 02 16:04:50.689476 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.689408    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8a6250-e0bf-4ab4-a3c5-084c93cbc028" containerName="revoke-keys"
Nov 02 16:04:50.692672 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.692648    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:50.731590 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.731541    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-hzcmf\" (UID: \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:50.731590 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.731576    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvbk5\" (UniqueName: \"kubernetes.io/projected/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-kube-api-access-pvbk5\") pod \"maas-api-revoke-sub-6b19338f-hzcmf\" (UID: \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:50.731870 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.731679    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-hzcmf\" (UID: \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:50.796878 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.796844    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 16:04:50.796878 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.796843    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:04:50.796878 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.796843    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 16:04:50.797144 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.796998    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7brzf\""
Nov 02 16:04:50.799435 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.799415    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"]
Nov 02 16:04:50.833061 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.833030    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-hzcmf\" (UID: \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:50.833238 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.833106    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-hzcmf\" (UID: \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:50.833238 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.833122    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvbk5\" (UniqueName: \"kubernetes.io/projected/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-kube-api-access-pvbk5\") pod \"maas-api-revoke-sub-6b19338f-hzcmf\" (UID: \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:50.833823 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.833800    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-hzcmf\" (UID: \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:50.835378 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.835360    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-hzcmf\" (UID: \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:50.894382 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:50.894346    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvbk5\" (UniqueName: \"kubernetes.io/projected/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-kube-api-access-pvbk5\") pod \"maas-api-revoke-sub-6b19338f-hzcmf\" (UID: \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:51.005729 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:51.005638    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:51.379996 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:51.379973    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"]
Nov 02 16:04:51.382237 ip-10-0-132-27 kubenswrapper[4891]: W1102 16:04:51.382201    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb30f1b_d507_4585_a3af_c8eeab9f0fc8.slice/crio-02f37fdd01e78b399fa8ff4e104d79ab06957518841e1f2e1222077a75bdb94f WatchSource:0}: Error finding container 02f37fdd01e78b399fa8ff4e104d79ab06957518841e1f2e1222077a75bdb94f: Status 404 returned error can't find the container with id 02f37fdd01e78b399fa8ff4e104d79ab06957518841e1f2e1222077a75bdb94f
Nov 02 16:04:51.970195 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:51.970094    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf" event={"ID":"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8","Type":"ContainerStarted","Data":"ff4b5e155420fcd09131a3465dbc927c59742023fac7d52752097c09ab4e4e39"}
Nov 02 16:04:51.970195 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:51.970175    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf" event={"ID":"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8","Type":"ContainerStarted","Data":"02f37fdd01e78b399fa8ff4e104d79ab06957518841e1f2e1222077a75bdb94f"}
Nov 02 16:04:52.091227 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:52.091171    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf" podStartSLOduration=2.091154672 podStartE2EDuration="2.091154672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:04:52.089534709 +0000 UTC m=+4099.682151509" watchObservedRunningTime="2026-11-02 16:04:52.091154672 +0000 UTC m=+4099.683771473"
Nov 02 16:04:52.978330 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:52.978283    4891 generic.go:356] "Generic (PLEG): container finished" podID="1cb30f1b-d507-4585-a3af-c8eeab9f0fc8" containerID="ff4b5e155420fcd09131a3465dbc927c59742023fac7d52752097c09ab4e4e39" exitCode=0
Nov 02 16:04:52.978794 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:52.978345    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf" event={"ID":"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8","Type":"ContainerDied","Data":"ff4b5e155420fcd09131a3465dbc927c59742023fac7d52752097c09ab4e4e39"}
Nov 02 16:04:54.126405 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.126371    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:54.167537 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.167504    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-kube-api-access-pvbk5\" (UniqueName: \"kubernetes.io/projected/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-kube-api-access-pvbk5\") pod \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\" (UID: \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\") "
Nov 02 16:04:54.167711 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.167577    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-openshift-ca\") pod \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\" (UID: \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\") "
Nov 02 16:04:54.167711 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.167599    4891 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-xks-ca\") pod \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\" (UID: \"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8\") "
Nov 02 16:04:54.167984 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.167955    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-openshift-ca" pod "1cb30f1b-d507-4585-a3af-c8eeab9f0fc8" (UID: "1cb30f1b-d507-4585-a3af-c8eeab9f0fc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:04:54.169753 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.169720    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-xks-ca" pod "1cb30f1b-d507-4585-a3af-c8eeab9f0fc8" (UID: "1cb30f1b-d507-4585-a3af-c8eeab9f0fc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:04:54.169874 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.169766    4891 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-kube-api-access-pvbk5" pod "1cb30f1b-d507-4585-a3af-c8eeab9f0fc8" (UID: "1cb30f1b-d507-4585-a3af-c8eeab9f0fc8"). InnerVolumeSpecName "kube-api-access-pvbk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:04:54.269370 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.269328    4891 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvbk5\" (UniqueName: \"kubernetes.io/projected/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-kube-api-access-pvbk5\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:04:54.269524 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.269376    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-openshift-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:04:54.269524 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.269393    4891 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb30f1b-d507-4585-a3af-c8eeab9f0fc8-maas-api-xks-ca\") on node \"ip-10-0-132-27.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 16:04:54.991216 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.991137    4891 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf"
Nov 02 16:04:54.991216 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.991171    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hzcmf" event={"ID":"1cb30f1b-d507-4585-a3af-c8eeab9f0fc8","Type":"ContainerDied","Data":"02f37fdd01e78b399fa8ff4e104d79ab06957518841e1f2e1222077a75bdb94f"}
Nov 02 16:04:54.991216 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:04:54.991215    4891 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02f37fdd01e78b399fa8ff4e104d79ab06957518841e1f2e1222077a75bdb94f"
Nov 02 16:05:03.051431 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:03.051394    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c96f6ccb5-dfw9m_dea5d7e8-792d-4444-9707-6e539828c206/authorino/0.log"
Nov 02 16:05:14.273522 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:05:14.273485    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:05:14.667636 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:14.667555    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dtrlb_c97de6e4-caef-4acb-b67e-04160d6d60ca/manager/0.log"
Nov 02 16:05:15.230105 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:15.230074    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fldg9_892e6d1a-7a1a-4f5f-bfac-456c20481bca/manager/0.log"
Nov 02 16:05:17.827865 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:17.827833    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-9m9sp_6c8a6250-e0bf-4ab4-a3c5-084c93cbc028/revoke-keys/0.log"
Nov 02 16:05:18.053956 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:18.053924    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-hzcmf_1cb30f1b-d507-4585-a3af-c8eeab9f0fc8/revoke-keys/0.log"
Nov 02 16:05:18.270887 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:18.270855    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-gjx87_a0cf2cd4-c831-427e-9fd1-b86357bd3e5b/revoke-keys/0.log"
Nov 02 16:05:19.951765 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:19.951718    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c96f6ccb5-dfw9m_dea5d7e8-792d-4444-9707-6e539828c206/authorino/0.log"
Nov 02 16:05:20.427866 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:20.427829    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hr5x2_12ce1ac0-a29b-4dde-867d-1ae8e712ff41/manager/0.log"
Nov 02 16:05:20.667335 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:20.667285    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9xdqd_8bdb971a-ff5a-4c55-aa72-0de03bb0a4f8/manager/0.log"
Nov 02 16:05:21.869073 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:21.869039    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dxtzb_113a5664-c697-4538-baf5-4aa46e2b54bc/manager/0.log"
Nov 02 16:05:22.702145 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:22.702105    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b65857h8ws_44c22dde-651b-4384-948c-d6467611acf5/istio-proxy/0.log"
Nov 02 16:05:23.430216 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:23.430171    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d9588f984-vxkqn_6eb5c56a-7799-400c-a789-bdb39d815648/kube-auth-proxy/0.log"
Nov 02 16:05:23.648633 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:23.648608    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-4lr42_8fbc97ef-c027-417f-aabd-a423f1d2f022/istio-proxy/0.log"
Nov 02 16:05:25.272851 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:25.272819    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-gv6hc_dc99f35f-221e-44ab-bd39-87d52079f398/istio-proxy/0.log"
Nov 02 16:05:25.546386 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:25.546294    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-j8c7c_4ecc4b7a-342a-4d11-ae0d-6731df9030aa/istio-proxy/0.log"
Nov 02 16:05:26.034977 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:26.034937    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p_3da03cab-4e6d-43fd-838c-332b9b6beb29/istio-proxy/0.log"
Nov 02 16:05:26.689328 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:26.689289    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-684675bd5f-zwwbp_1bcba632-1434-4533-951f-84b8bba1f0d8/router/0.log"
Nov 02 16:05:30.168004 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:30.167967    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c96f6ccb5-dfw9m_dea5d7e8-792d-4444-9707-6e539828c206/authorino/0.log"
Nov 02 16:05:39.779742 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:39.779706    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dtrlb_c97de6e4-caef-4acb-b67e-04160d6d60ca/manager/0.log"
Nov 02 16:05:40.368746 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:40.368713    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fldg9_892e6d1a-7a1a-4f5f-bfac-456c20481bca/manager/0.log"
Nov 02 16:05:42.682190 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:42.682156    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-9m9sp_6c8a6250-e0bf-4ab4-a3c5-084c93cbc028/revoke-keys/0.log"
Nov 02 16:05:42.915139 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:42.915111    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-hzcmf_1cb30f1b-d507-4585-a3af-c8eeab9f0fc8/revoke-keys/0.log"
Nov 02 16:05:43.139940 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:43.139914    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-gjx87_a0cf2cd4-c831-427e-9fd1-b86357bd3e5b/revoke-keys/0.log"
Nov 02 16:05:44.272920 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:05:44.272883    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:05:44.558156 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:44.558057    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c96f6ccb5-dfw9m_dea5d7e8-792d-4444-9707-6e539828c206/authorino/0.log"
Nov 02 16:05:45.035816 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:45.035782    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hr5x2_12ce1ac0-a29b-4dde-867d-1ae8e712ff41/manager/0.log"
Nov 02 16:05:45.274723 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:45.274685    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9xdqd_8bdb971a-ff5a-4c55-aa72-0de03bb0a4f8/manager/0.log"
Nov 02 16:05:46.406225 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:46.406184    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dxtzb_113a5664-c697-4538-baf5-4aa46e2b54bc/manager/0.log"
Nov 02 16:05:47.036165 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:47.036130    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b65857h8ws_44c22dde-651b-4384-948c-d6467611acf5/istio-proxy/0.log"
Nov 02 16:05:47.831492 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:47.831445    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d9588f984-vxkqn_6eb5c56a-7799-400c-a789-bdb39d815648/kube-auth-proxy/0.log"
Nov 02 16:05:48.047975 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:48.047943    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-4lr42_8fbc97ef-c027-417f-aabd-a423f1d2f022/istio-proxy/0.log"
Nov 02 16:05:49.858512 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:49.858481    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-gv6hc_dc99f35f-221e-44ab-bd39-87d52079f398/istio-proxy/0.log"
Nov 02 16:05:50.131922 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:50.131836    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-j8c7c_4ecc4b7a-342a-4d11-ae0d-6731df9030aa/istio-proxy/0.log"
Nov 02 16:05:50.634917 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:50.634883    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-kgs4p_3da03cab-4e6d-43fd-838c-332b9b6beb29/istio-proxy/0.log"
Nov 02 16:05:51.301485 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:05:51.301451    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-684675bd5f-zwwbp_1bcba632-1434-4533-951f-84b8bba1f0d8/router/0.log"
Nov 02 16:06:14.273002 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:06:14.272915    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:06:33.487634 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:06:33.487606    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 16:06:33.488664 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:06:33.488640    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 16:06:33.499619 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:06:33.499596    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctb2z4_1cf1bed8-3a81-47af-ad2a-d401fc22d0dc/extract/0.log"
Nov 02 16:06:33.500604 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:06:33.500584    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqwml_e1cb9fa4-4a58-459d-9bfc-09d650cf3657/extract/0.log"
Nov 02 16:06:33.503509 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:06:33.503493    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 16:06:33.515132 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:06:33.515112    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 16:06:44.273670 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:06:44.273629    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:07:14.273447 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:07:14.273404    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:07:19.002143 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:19.002085    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nx6px_41cff577-8a79-4b9e-89c8-a93e1ecf4ff0/global-pull-secret-syncer/0.log"
Nov 02 16:07:20.255123 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:20.255094    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-64bm4_63faa28d-8806-4332-99c4-9f715a0b1a86/connection-checker/0.log"
Nov 02 16:07:20.696325 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:20.696286    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-v4f7p_f25b0184-4053-4694-acc1-c26e9f5602aa/connection-checker/0.log"
Nov 02 16:07:21.137932 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:21.137904    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xx5tn_0290bc25-3e34-4d43-9215-f15cfc44f686/connection-checker/0.log"
Nov 02 16:07:22.047171 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:22.047134    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-spbjm_c7d93f72-6fc0-469a-8c16-9feed09bda86/konnectivity-agent/0.log"
Nov 02 16:07:23.003667 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:23.003638    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-27.us-west-1.compute.internal_c52ee3e32e488dfc59e1227cdb8da56d/haproxy/0.log"
Nov 02 16:07:27.908418 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:27.908386    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c96f6ccb5-dfw9m_dea5d7e8-792d-4444-9707-6e539828c206/authorino/0.log"
Nov 02 16:07:28.226914 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:28.226814    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hr5x2_12ce1ac0-a29b-4dde-867d-1ae8e712ff41/manager/0.log"
Nov 02 16:07:28.644194 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:28.644165    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9xdqd_8bdb971a-ff5a-4c55-aa72-0de03bb0a4f8/manager/0.log"
Nov 02 16:07:30.604445 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:30.604413    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dxtzb_113a5664-c697-4538-baf5-4aa46e2b54bc/manager/0.log"
Nov 02 16:07:36.197891 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:36.197857    4891 scope.go:122] "RemoveContainer" containerID="1f5d0133e0f529e4f2e6f00b2948bdc5a2077538653848e14e4847b30eaa31cd"
Nov 02 16:07:36.262289 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:36.262232    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-q2t8x_885145c5-6077-4435-89ce-73af1e6a5600/cluster-monitoring-operator/0.log"
Nov 02 16:07:40.104138 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:40.104104    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2p4h_170d86eb-a48d-4125-b264-fff55074f210/node-exporter/0.log"
Nov 02 16:07:40.519608 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:40.519581    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2p4h_170d86eb-a48d-4125-b264-fff55074f210/kube-rbac-proxy/0.log"
Nov 02 16:07:40.940578 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:40.940498    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2p4h_170d86eb-a48d-4125-b264-fff55074f210/init-textfile/0.log"
Nov 02 16:07:44.273509 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:07:44.273429    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:07:48.247565 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.247531    4891 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"]
Nov 02 16:07:48.248065 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.247941    4891 cpu_manager.go:414] "containerMap: removing container" podUID="1cb30f1b-d507-4585-a3af-c8eeab9f0fc8" containerName="revoke-keys"
Nov 02 16:07:48.248065 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.247955    4891 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb30f1b-d507-4585-a3af-c8eeab9f0fc8" containerName="revoke-keys"
Nov 02 16:07:48.248065 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.248024    4891 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb30f1b-d507-4585-a3af-c8eeab9f0fc8" containerName="revoke-keys"
Nov 02 16:07:48.251133 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.251116    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.299866 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.299832    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlhg\" (UniqueName: \"kubernetes.io/projected/1b8270b5-274b-47dc-8938-c9c392a770d0-kube-api-access-tdlhg\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.299866 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.299877    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b8270b5-274b-47dc-8938-c9c392a770d0-sys\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.300073 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.299899    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b8270b5-274b-47dc-8938-c9c392a770d0-podres\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.300073 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.299962    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b8270b5-274b-47dc-8938-c9c392a770d0-lib-modules\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.300150 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.300074    4891 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b8270b5-274b-47dc-8938-c9c392a770d0-proc\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.341162 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.341117    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"]
Nov 02 16:07:48.392900 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.392864    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gjv5q\"/\"kube-root-ca.crt\""
Nov 02 16:07:48.401496 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.401471    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlhg\" (UniqueName: \"kubernetes.io/projected/1b8270b5-274b-47dc-8938-c9c392a770d0-kube-api-access-tdlhg\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.401591 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.401508    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b8270b5-274b-47dc-8938-c9c392a770d0-sys\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.401591 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.401528    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b8270b5-274b-47dc-8938-c9c392a770d0-podres\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.401708 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.401593    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b8270b5-274b-47dc-8938-c9c392a770d0-sys\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.401708 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.401635    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b8270b5-274b-47dc-8938-c9c392a770d0-lib-modules\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.401708 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.401668    4891 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b8270b5-274b-47dc-8938-c9c392a770d0-proc\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.401708 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.401683    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b8270b5-274b-47dc-8938-c9c392a770d0-podres\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.401896 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.401777    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b8270b5-274b-47dc-8938-c9c392a770d0-lib-modules\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.401896 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.401798    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b8270b5-274b-47dc-8938-c9c392a770d0-proc\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.413992 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.413956    4891 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gjv5q\"/\"openshift-service-ca.crt\""
Nov 02 16:07:48.413992 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.413969    4891 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gjv5q\"/\"default-dockercfg-2qgwv\""
Nov 02 16:07:48.462194 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.462167    4891 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlhg\" (UniqueName: \"kubernetes.io/projected/1b8270b5-274b-47dc-8938-c9c392a770d0-kube-api-access-tdlhg\") pod \"perf-node-gather-daemonset-cpmnw\" (UID: \"1b8270b5-274b-47dc-8938-c9c392a770d0\") " pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.563977 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.563936    4891 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:48.944585 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.944553    4891 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"]
Nov 02 16:07:48.946464 ip-10-0-132-27 kubenswrapper[4891]: W1102 16:07:48.946434    4891 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1b8270b5_274b_47dc_8938_c9c392a770d0.slice/crio-884bc15148f2b238e6cc3c26ca8a4dc534afcb13b67bcbf456a126c3afa52a51 WatchSource:0}: Error finding container 884bc15148f2b238e6cc3c26ca8a4dc534afcb13b67bcbf456a126c3afa52a51: Status 404 returned error can't find the container with id 884bc15148f2b238e6cc3c26ca8a4dc534afcb13b67bcbf456a126c3afa52a51
Nov 02 16:07:48.995470 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:48.995437    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw" event={"ID":"1b8270b5-274b-47dc-8938-c9c392a770d0","Type":"ContainerStarted","Data":"884bc15148f2b238e6cc3c26ca8a4dc534afcb13b67bcbf456a126c3afa52a51"}
Nov 02 16:07:50.002547 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:50.002506    4891 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw" event={"ID":"1b8270b5-274b-47dc-8938-c9c392a770d0","Type":"ContainerStarted","Data":"43626c900352984767ad944302c2aa83fd1c4109a2089220bcc05ec906668a05"}
Nov 02 16:07:50.003051 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:50.002580    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:50.120710 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:50.120647    4891 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw" podStartSLOduration=2.120634897 podStartE2EDuration="2.120634897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 16:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:07:50.117566654 +0000 UTC m=+4277.710183455" watchObservedRunningTime="2026-11-02 16:07:50.120634897 +0000 UTC m=+4277.713251696"
Nov 02 16:07:54.247663 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:54.247633    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-mf5tv_fb39336d-67c0-4cf2-aaff-c873d361f9b2/networking-console-plugin/0.log"
Nov 02 16:07:56.018567 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:56.018534    4891 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gjv5q/perf-node-gather-daemonset-cpmnw"
Nov 02 16:07:56.937756 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:56.937727    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/2.log"
Nov 02 16:07:56.941946 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:07:56.941929    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rtsrv_bf04dbe2-4354-4849-959c-33002c7ece68/console-operator/3.log"
Nov 02 16:08:02.529391 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:08:02.529358    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-96v9l_27c6485e-f8bc-4b94-839c-746297629798/volume-data-source-validator/0.log"
Nov 02 16:08:08.793778 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:08:08.793746    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f6fgv_b639a9d2-f451-4cb1-a31f-c3de10b8d3c1/dns/0.log"
Nov 02 16:08:09.212816 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:08:09.212741    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f6fgv_b639a9d2-f451-4cb1-a31f-c3de10b8d3c1/kube-rbac-proxy/0.log"
Nov 02 16:08:10.885145 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:08:10.885111    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cllms_74049a08-892a-4ae2-a5a5-161da6182c1c/dns-node-resolver/0.log"
Nov 02 16:08:14.274557 ip-10-0-132-27 kubenswrapper[4891]: E1102 16:08:14.274514    4891 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtxzd" containerName="extractor"
Nov 02 16:08:14.627988 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:08:14.627910    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9htnf_acede18e-4d8e-47ee-b31f-918658a9ffba/node-ca/0.log"
Nov 02 16:08:19.905737 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:08:19.905705    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b65857h8ws_44c22dde-651b-4384-948c-d6467611acf5/istio-proxy/0.log"
Nov 02 16:08:21.809568 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:08:21.809535    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d9588f984-vxkqn_6eb5c56a-7799-400c-a789-bdb39d815648/kube-auth-proxy/0.log"
Nov 02 16:08:22.231099 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:08:22.231017    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-4lr42_8fbc97ef-c027-417f-aabd-a423f1d2f022/istio-proxy/0.log"
Nov 02 16:08:25.200187 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:08:25.200154    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-gv6hc_dc99f35f-221e-44ab-bd39-87d52079f398/istio-proxy/0.log"
Nov 02 16:08:25.623364 ip-10-0-132-27 kubenswrapper[4891]: I1102 16:08:25.623332    4891 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-j8c7c_4ecc4b7a-342a-4d11-ae0d-6731df9030aa/istio-proxy/0.log"